Information processing apparatus, information processing method, and information processing system

ABSTRACT

An information processing apparatus includes an acquisition unit that acquires plural pieces of medical apparatus information which are plural pieces of information acquired from a plurality of medical apparatuses; a generation unit that generates presentation information which contains at least a part of the plural pieces of medical apparatus information acquired by the acquisition unit; and a presentation unit that outputs the generated presentation information to a plurality of image display apparatuses.

TECHNICAL FIELD

The present invention relates to an information processing apparatus.

BACKGROUND ART

There is known a head mounted display apparatus (a head mounted display(HMD)) which is a display apparatus mounted on the head. For example,the head mounted display apparatus generates image light for an imageusing a liquid crystal display and a light source, and leads thegenerated image light to the eyes of a user using a projection opticalsystem, a light guide plate, or the like, and thus the user is allowedto recognize a virtual image.

PTL 1 discloses a system in which the head mounted display apparatus isused, which improves display of information in a medical environment. Inthe technology disclosed in PTL 1, information is acquired from aplurality of information sources based on a query. The acquiredinformation undergoes a filtering process based on rules and isprojected on eyepieces of a headset-type eyeglass viewer. In thetechnology disclosed in PTL 1, a control right and a priority right fordisplay of the information on a fixed large display are determined basedon a visual line of a person that wears the head mounted displayapparatus. PTL 2 discloses a face mounted video display apparatus formedical use (a head mounted display apparatus). In the technologydisclosed in PTL 2, operation state information of peripheral devices ofan endoscope, which are used with the endoscope for an endoscopicexamination or an endoscopic surgery, is displayed on the head mounteddisplay apparatus. PTL 3 and PTL 4 disclose a method in which anaugmented reality technology is applied to display information of adental surgery in the head mounted display apparatus. In the technologydisclosed in PTL 3 and PTL 4, display of a preoperative image, asimulated image, or the like for an operated part is superimposed on anactual image of an operated part.

CITATION LIST Patent Literature

[PTL 1]

JP-A-2006-102495

[PTL 2]

JP-A-2001-104331

[PTL 3]

JP-A-2010-259497

[PTL 4]

JP-A-2013-34764

SUMMARY OF INVENTION Technical Problem

Progresses in Medicine and electronic devices have brought about rapidprogress in current medical locations. In the medical locations, variousapparatuses for medical use (hereinafter, referred to as “medicalapparatuses”) such as an electrocardiogram, a heart rate meter, a bloodpressure meter, a pulse oximeter, a blood glucose meter, an X-ray, andan endoscope are used. It is a great burden for those in the medicallocations to understand information of the various medical apparatusesand to perform tasks. For this reason, there has been a demand for asystem in which all the information of the various medical apparatusescan be unified. In the technology disclosed in PTL 1, information can beacquired from a plurality of information sources (medical apparatuses),but there is no mention of how the acquired information is displayed ona plurality of the head mounted display apparatuses. In the technologiesdisclosed in PTL 2 to PTL 4, targeted medical apparatuses are limited,and thus there is a problem in that the information of the variousmedical apparatuses cannot be unified.

The problem is not limited to the head mounted display apparatus, but iscommon to all information processing apparatuses that generateinformation to be displayed on an image display apparatus such as thehead mounted display apparatus. As such, there is a demand for aninformation processing apparatus that can unify the information of thevarious medical apparatuses, and that can control a display of theinformation on a plurality of image display apparatuses. In addition,there are various demands for the information processing apparatus asfollows: a versatility improvement, a convenience improvement, amanufacturing cost reduction, and a reliability improvement.

Solution to Problem

An advantage of some aspects of the invention is to solve at least apart of the problems described above, and the invention can beimplemented as the following aspects.

(1) An aspect of the invention provides an information processingapparatus. The information processing apparatus includes an acquisitionunit that acquires plural pieces of medical apparatus information whichare plural pieces of information acquired from a plurality of medicalapparatuses; a generation unit that generates plural pieces ofpresentation information which respectively contain at least a part ofthe plural pieces of medical apparatus information acquired by theacquisition unit; and a presentation unit that outputs the generatedpresentation information to a plurality of image display apparatuses. Inthe information processing apparatus according to the aspect, thegeneration unit generates the presentation information using the pluralpieces of medical apparatus information acquired by the acquisitionunit, and thus the information (the medical apparatus information) ofvarious medical apparatuses which are connected to the informationprocessing apparatus can be unified. The presentation unit outputs thegenerated presentation information to the plurality of image displayapparatuses, and thus the generation unit can control display on theplurality of image display apparatuses. As a result, it is possible tounify the information of various medical apparatuses, and it is possibleto realize the information processing apparatus that can control thedisplay on the plurality of image display apparatuses.

(2) The information processing apparatus according to the aspectdescribed above may be configured such that the generation unit has afirst mode in which the presentation information common to the pluralityof image display apparatuses is generated, and a second mode in whichthe plural pieces of presentation information are generated torespectively correspond to the plurality of image display apparatuses.In the information processing apparatus according to this aspect, thegeneration unit has the first mode in which the presentation informationcommon to the plurality of image display apparatuses is generated usingthe unified medical apparatus information, and the second mode in whichthe plural pieces of presentation information are generated torespectively correspond to the plurality of image display apparatuses.The generation unit uses the modes separately, and thus the informationprocessing apparatus can control the display on the plurality of imagedisplay apparatuses.

(3) The information processing apparatus according to the aspectdescribed above may be configured such that, in the first mode, thegeneration unit generates the common presentation information thatcontains at least the medical apparatus information selected by a userof a specific image display apparatus. In the information processingapparatus according to this aspect, the generation unit generates thecommon presentation information that contains at least the medicalapparatus information selected by the user of the specific image displayapparatus, and the presentation unit outputs the generated commonpresentation information to the plurality of image display apparatuses.As a result, the information processing apparatus can control display onthe plurality of image display apparatuses based on the selection of theuser of the specific image display apparatus.

(4) The information processing apparatus according to the aspectdescribed above may be configured such that the information processingapparatus further includes an inter-apparatus relationship storage unitthat stores information for individually identifying the plurality ofimage display apparatuses in association with information for indicatingwhether the image display apparatus is the specific image displayapparatus, and in the first mode, the generation unit identifies thespecific image display apparatus using the inter-apparatus relationshipstorage unit. In the information processing apparatus according to thisaspect, the generation unit can simply identify the specific imagedisplay apparatus from the plurality of image display apparatuses usingthe inter-apparatus relationship storage unit.

(5) The information processing apparatus according to the aspectdescribed above may be configured such that, in the second mode, thegeneration unit generates the plural pieces of presentation informationthat respectively contain at least the pieces of acquired medicalapparatus information which are respectively selected by users of theplurality of image display apparatuses. In the information processingapparatus according to this aspect, the generation unit generates theplural pieces of presentation information that respectively contain atleast the pieces of medical apparatus information which are respectivelyselected by the users of the plurality of image display apparatuses. Thepresentation unit outputs the plural pieces of generated presentationinformation to the plurality of image display apparatuses. As a result,the information processing apparatus can individually control display onthe plurality of image display apparatuses based on a selection of eachuser of the image display apparatus.

(6) The information processing apparatus according to the aspectdescribed above may be configured such that, in the second mode, thegeneration unit generates the plural pieces of presentation informationthat respectively contain at least the pieces of medical apparatusinformation acquired from the medical apparatuses based on the role ofeach user. In the information processing apparatus according to thisaspect, the generation unit generates the plural pieces of presentationinformation that respectively contain at least the pieces of medicalapparatus information which are acquired from the medical apparatusesbased on the role of each user. The presentation unit outputs the pluralpieces of generated presentation information to the plurality of imagedisplay apparatuses. As a result, the information processing apparatuscan individually control display on the plurality of image displayapparatuses based on the role of each user of the image displayapparatus, and it is possible to improve convenience of each user of theimage display apparatus.

(7) The information processing apparatus according to the aspectdescribed above may be configured such that the information processingapparatus further includes a role storage unit that stores theinformation for individually identifying the plurality of image displayapparatuses in association with a role of a user of the image displayapparatus, and in the second mode, the generation unit identifies therole of the user using the role storage unit. In the informationprocessing apparatus according to this aspect, the generation unit cansimply identify a role of each user using the role storage unit.

(8) The information processing apparatus according to the aspectdescribed above may be configured such that, in the second mode, thegeneration unit further generates combined information in which theplural pieces of generated presentation information are combinedtogether, and the presentation unit outputs the combined informationthat is generated for the specific image display apparatus. In theinformation processing apparatus according to this aspect, thegeneration unit generates the combined information in which the pluralpieces of generated presentation information are combined together, andthe presentation unit outputs the combined information that is generatedfor the specific image display apparatus. As a result, the user of thespecific image display apparatus can understand what presentationinformation is displayed on the plurality of image display apparatusesconnected to the information processing apparatus. As a result, it ispossible to improve convenience of the user of the specific imagedisplay apparatus.

(9) The information processing apparatus according to the aspectdescribed above may be configured such that, when the generation unitgenerates the presentation information, the generation unit enlarges orreduce an image while maintaining an aspect ratio of the image that iscontained in the medical apparatus information. In the informationprocessing apparatus according to this aspect, the generation unitenlarges or reduces the image while maintaining the aspect ratio of theimage that is contained in the medical apparatus information, and thusit is possible to suppress unexpected distortion of the medicalapparatus information acquired by the medical apparatus when thepresentation information is generated.

(10) The information processing apparatus according to the aspectdescribed above may be configured such that the information processingapparatus is connected to a head mounted display apparatus as the imagedisplay apparatus, on which a user can see a virtual image and anoutside scene. In the information processing apparatus according to thisaspect, it is possible to unify the information of various medicalapparatuses, and it is possible to realize the information processingapparatus that can control display on the plurality of head mounteddisplay apparatuses.

(11) The information processing apparatus according to the aspectdescribed above may be configured such that the presentation unitfurther acquires from the head mounted display apparatus an image in adirection of a visual field of a user that wears the head mounteddisplay apparatus, the presentation unit further extractscharacteristics of an object contained in the outside scene byrecognizing the image acquired in the direction of the visual field, andthe presentation unit further compensates the presentation informationgenerated by the generation unit based on the extracted characteristics,and outputs the compensated presentation information. In the informationprocessing apparatus according to this aspect, the presentation unitextracts the characteristics of the object contained in the outer sceneby recognizing the image in the direction of the visual field of theuser of the head mounted display apparatus, and the presentation unitcompensates the generated presentation information based on theextracted characteristics. For this reason, when the presentationinformation contains invisible part information such as a CT image, anMRI image, or an X-ray image, it is possible to align the invisible partinformation with the object (for example, the body of a patient)contained in the outside scene. As a result, it is possible to improveconvenience of the user of the head mounted display apparatus.Furthermore, since the head mounted display apparatus can receive thecompensated presentation information, the head mounted display apparatusmay display the received presentation information as it is. As a result,it is possible to simplify a configuration of the head mounted displayapparatus.

(12) The information processing apparatus according to the aspectdescribed above may be configured such that the information processingapparatus further includes a step management unit that transmitsinformation on a predetermined step to the generation unit, and thatchanges the step to be transmitted based on the plural pieces of medicalapparatus information acquired by the acquisition unit, and thegeneration unit generates the presentation information that contains thereceived step. In the information processing apparatus according to thisaspect, the step management unit transmits the information on thepredetermined step to the generation unit, and the generation unitgenerates the presentation information that contains the received step.For this reason, the information processing apparatus can display thepredetermined step on the plurality of image display apparatusesconnected to the information processing apparatus. The step managementunit changes the step to be transmitted based on the plural pieces ofmedical apparatus information that are acquired by the acquisition unit.For this reason, the information processing apparatus can change thestep to be displayed on the plurality of image display apparatuses basedon the plural pieces of medical apparatus information that are acquiredby the plurality of medical apparatuses. As a result, it is possible toprovide the information processing apparatus that can flexibly changethe predetermined step based on the medical apparatus information, andcan guide each user of the image display apparatus through the step.

(13) The information processing apparatus according to the aspectdescribed above may be configured such that the step management unitrepeatedly acquires the plural pieces of medical apparatus information,and changes the step to be transmitted to the generation unit based onthe plural pieces of latest medical apparatus information. In theinformation processing apparatus according to this aspect, the stepmanagement unit repeatedly acquires the plural pieces of medicalapparatus information, and changes the step to be transmitted to thegeneration unit based on the plural pieces of latest medical apparatusinformation. For this reason, the information processing apparatus canchange the step to be displayed on the plurality of image displayapparatuses in real time based on the plural pieces of medical apparatusinformation that are acquired from the plurality of medical apparatuses.

(14) The information processing apparatus according to the aspectdescribed above may be configured such that the step management unittransmits to the generation unit information on a series ofpredetermined steps which are formed of a plurality of steps, and thestep management unit changes the steps by performing at least one of thefollowing procedures: addition of a new step to the series of steps,removal of at least a part of steps from the series of steps, andreplacement of at least a part of steps in the series of steps. In theinformation processing apparatus according to this aspect, the stepmanagement unit may change the steps by performing at least one of thefollowing procedures: addition of a new step to the series of steps,removal of at least a part of steps from the series of steps, andreplacement of at least a part of steps in the series of steps.

(15) The information processing apparatus according to this aspectdescribed above may be configured such that the step management unitsequentially transmits to the generation unit the series ofpredetermined steps one by one, which are formed of the plurality ofsteps and the step management unit changes the steps by stopping thetransmission of the single step. In the information processing apparatusaccording to this aspect, the step management unit can change the stepsby stopping the transmission of the step.

(16) The information processing apparatus according to the aspectdescribed above may be configured such that, in the first mode, the stepmanagement unit transmits the steps for the user of the specific imagedisplay apparatus to the generation unit. In the information processingapparatus according to this aspect, in the first mode, the stepmanagement unit transmits the steps for the user of the specific imagedisplay apparatus to the generation unit. For this reason, theinformation processing apparatus can display common “steps for the userof the specific image display apparatus” on the plurality of imagedisplay apparatuses.

(17) The information processing apparatus according to the aspectdescribed above may be configured such that, in the second mode, thestep management unit transmits to the generation unit the steps thatrespectively correspond to the roles of the users. In the informationprocessing apparatus according to this aspect, in the second mode, thestep management unit transmits to the generation unit the steps thatrespectively correspond to the roles of the users. For this reason, theinformation processing apparatus can individually display the “stepswhich respectively correspond to the roles of the users of the imagedisplay apparatuses” on the plurality of image display apparatuses.

(18) Another aspect of the invention provides an information processingsystem. The information processing system includes a plurality ofmedical apparatuses; an information processing apparatus; and aplurality of head mounted display apparatuses, on each of which a usercan see a virtual image and an outside scene. The information processingapparatus includes an acquisition unit that acquires plural pieces ofmedical apparatus information that are acquired from the plurality ofmedical apparatuses; a generation unit that generates presentationinformation which contains at least a part of the plural pieces ofmedical apparatus information acquired by the acquisition unit; and apresentation unit that outputs the generated presentation information tothe plurality of head mounted display apparatuses. The head mounteddisplay apparatus includes a presentation information acquisition unitthat acquires the presentation information from the informationprocessing apparatus; and an image display unit on which a user of thehead mounted display apparatus sees the acquired presentationinformation as the virtual image. In the information processing systemaccording to the aspect, it is possible to unify information of variousmedical apparatus (medical apparatus information) using the informationprocessing apparatus, and it is possible to realize the informationprocessing system in which display on the plurality of head mounteddisplay apparatuses can be controlled.

Not the entirety of a plurality of configuration elements of each aspectof the invention is essential. It is possible to appropriately change,remove, replace a part of the plurality of configuration elements withnew configuration elements, or to remove a part of limited content so asto solve a part or the entirety of the problems, or to achieve a part orthe entirety of the advantages described in this specification. A partor the entirety of technical characteristics contained in one aspect ofthe invention can be combined with a part or the entirety of technicalcharacteristics contained in the other aspects of the invention, or canbe an independent aspect of the invention so as to solve a part or theentirety of the problems, or to achieve a part or the entirety of theadvantages described in this specification.

For example, an aspect of the invention can be realized as an apparatusthat includes a part or the entirety of three configuration elementswhich are the acquisition unit, the generation unit, and thepresentation unit. That is, the apparatus may or may not include theacquisition unit. The apparatus may or may not include the generationunit. The apparatus may or may not include the presentation unit. Forexample, the apparatus can be realized as the information processingapparatus, but the apparatus can also be realized as apparatuses otherthan the information processing apparatus. A part or the entirety of thetechnical characteristics of each aspect of the information processingapparatus can be applied to any apparatus.

The invention can be realized in various forms. For example, it ispossible to realize the invention in the following forms: theinformation processing apparatus; the information processing method; theinformation processing system; the head mounted display apparatus; amethod of controlling the head mounted display apparatus; a head mounteddisplay system; a computer program for the realization of functions ofthe methods, the apparatuses, and the systems, a record medium on whichthe computer program is recorded, and the like.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a schematic configuration of aninformation processing system according to an embodiment of theinvention.

FIG. 2 is a block diagram illustrating a functional configuration of aninformation processing apparatus.

FIG. 3 is a table illustrating an example of a common videospecification.

FIG. 4 is a table illustrating an example of a management table.

FIG. 5 is a block diagram illustrating a functional configuration of ahead mounted display.

FIG. 6 is a flow chart illustrating procedures of presentationinformation generating process.

FIG. 7A illustrates a view describing a method of generating thepresentation information.

FIG. 7B illustrates a view describing a method of generating thepresentation information.

FIG. 8 is a view describing the method of generating the presentationinformation.

FIG. 9A illustrates a view showing a first example of the presentationinformation.

FIG. 9B illustrates a view showing a first example of the presentationinformation.

FIG. 10A illustrates a view showing a second example of the presentationinformation.

FIG. 10B illustrates a view showing a second example of the presentationinformation.

FIG. 11 is a view illustrating a third example of the presentationinformation.

FIG. 12 is a view illustrating a state in which the presentationinformation is displayed as a virtual image on a plurality of headmounted displays.

FIG. 13 is a view illustrating a state in which the presentationinformation is displayed as a virtual image on the plurality of headmounted displays that are positioned in different places.

FIG. 14 is a view illustrating a state in which the presentationinformation is displayed as a virtual image on the plurality of headmounted displays after steps S122 to S134 are executed two times.

FIG. 15 is a block diagram illustrating a functional configuration of aninformation processing apparatus according to Second Embodiment.

FIG. 16 is a flow chart illustrating procedures of combined informationgenerating process.

FIG. 17 is a view describing a method of generating the combinedinformation.

FIG. 18 is a view illustrating a state in which the combined informationis displayed as a virtual image on the head mounted display.

FIG. 19 is a block diagram illustrating a functional configuration of aninformation processing apparatus according to Third Embodiment.

FIG. 20 is a schematic diagram illustrating an example of stepinformation.

FIG. 21 is a table illustrating an example of emergency responseinformation.

FIG. 22 is a flow chart illustrating procedures of a step managementprocess.

FIG. 23A illustrates a view showing an external configuration of eachhead mounted display in modification examples.

FIG. 23B illustrates a view showing an external configuration of eachhead mounted display in modification examples.

DESCRIPTION OF EMBODIMENTS A. First Embodiment

A-1. Configuration of Information Processing System:

FIG. 1 is a diagram illustrating a schematic configuration of aninformation processing system 1000 according to an embodiment of theinvention. The information processing system 1000 includes a pluralityof medical apparatuses 501 to 50 n, an information processing apparatus300, and an image display apparatus 100. The information processingsystem 1000 is a system used in a medical location. The informationprocessing apparatus 300 acquires information from the plurality ofmedical apparatuses 501 to 50 n, unifies the information, and generatespresentation information to be displayed on the image display apparatus100. The medical apparatuses 501 to 50 n and the image display apparatus100 are connected to the information processing apparatus 300 by a wiredconnection or a wireless connection. Hereinafter, “n” is used as aletter indicating an “arbitrary plural”. n is an integer equal to orgreater than 1.

In the embodiment, an operation site in an operation room is illustratedas the medical location. However, the information processing system 1000may be used in a consulting room, a treatment room, a curing room, adelivery room, and the like. In the embodiment, a head mounted displayapparatus is illustrated as the image display apparatus 100. However,the image display apparatus 100 may be a stationary display, a portabledisplay, a smart phone, and the like. Hereinafter, the head mounteddisplay apparatus 100 is also referred to as the “head mounted display(HMD) 100”. In FIG. 1, a single head mounted display 100 is illustrated.However, as illustrated in FIG. 2 to be described below, a plurality ofhead mounted displays 100 a to 100 n may be connected to the informationprocessing apparatus 300.

In the following description, when the head mounted displays 100 a to100 n are not particularly specified and are generally described, thehead mounted displays 100 a to 100 n are simply referred to as the “headmounted display 100”. Similarly, when the medical apparatuses 501 to 50n are not particularly specified and are generally described, themedical apparatuses 501 to 50 n are simply referred to as a “medicalapparatus 500”.

The medical apparatuses 501 to 50 n are various apparatuses for amedical use. For example, the following can be used as the medicalapparatuses 501 to 50 n: a visible light camera for photographing indoorand outdoor still images and indoor and outdoor moving images, a visiblelight camera for photographing still and moving images of an operatedpart, an electrocardiogram examination apparatus, a timer for surgery, abiological monitor, a heart rate meter, a pulse rate meter, a bloodpressure meter, an ultrasonic examination apparatus, a pulse oximeter, ablood glucose meter, a CT (Computed Tomography) image photographingapparatus, an MRI (Magnetic Resonance Imaging) image photographingapparatus, an X-ray image photographing apparatus, a fluoroscopic imagephotographing apparatus, an endoscopic image photographing apparatus, athermography image photographing apparatus, and the like.

A-2. Configuration of Information Processing Apparatus:

FIG. 2 is a block diagram illustrating a functional configuration of theinformation processing apparatus 300. The information processingapparatus 300 includes a first input/output unit 310, a ROM 320, a RAM330, a communication interface 340, a CPU 350, and an external storageapparatus 380. The units are connected to each other via a bus.

The first input/output unit 310 as an acquisition unit includes aplurality of first input/output units 311 to 31 n. For example, thefirst input/output units 311 to 31 n are the following input/outputinterfaces conforming to a communication standard: a USB (UniversalSerial Bus), an HDMI (High Definition Multimedia Interface), a DVI(Digital Visual Interface), a VGA (Video Graphics Array), a composite,an RS-232C (Recommended Standard 232), an infrared ray, a short-rangewireless communication (for example, Bluetooth (registered trademark)),a wireless LAN (for example, IEEE802.11), a wired LAN (for example,IEEE802.3), and the like. The first input/output units 311 to 31 nrespectively acquire information input from the medical apparatuses 501to 50 n that are respectively connected thereto. The information inputfrom the medical apparatus is information measured by or photographed bythe medical apparatus. Hereinafter, the information is also referred toas “medical apparatus information”.

In the embodiment, the first input/output unit 311 is an input/outputinterface conforming to the DVI. The first input/output unit 312 is aninput/output interface conforming to the HDMI. The first input/outputunit 313 is an input/output interface conforming to the HDMI. The firstinput/output unit 31 n is an input/output interface conforming to thewireless LAN. In FIG. 2, the medical apparatus 501 and a stationarydisplay 700 are connected to the first input/output unit 311. Similarly,the medical apparatus 502 is connected to the first input/output unit312, the medical apparatus 503 is connected to the first input/outputunit 313, and the medical apparatus 50 n is connected to the firstinput/output unit 31 n. In this way, the first input/output unit 310 isconfigured to have the first input/output units 311 to 31 n conformingto the different types of communication standards. For this reason, itis possible to connect the medical apparatuses conforming to variouscommunication standards and the first input/output unit 310, and it ispossible to improve versatility of the information processing apparatus300. The first input/output unit 310 can also output to the displayapparatus such as the display 700 an input signal as it is.

The communication interface (I/F) 340 is a wired LAN interface. Thecommunication interface 340 communicatively connects a doctor PR in anexternal facility and the information processing apparatus 300 via anInternet INT line. The communication interface 340 communicativelyconnects a database OD connected to an in-hospital LAN and theinformation processing apparatus 300 via an intranet. For example, thedoctor PR is assumed to be a doctor who supervises a user of the headmounted display 100 that is connected to the information processingapparatus 300, or is assumed to be an outside expert. The database ODstores procedure information illustrating a procedure of a treatmentsuch as an operation, an electronic medical record system, electronicacademic books, and the like.

The CPU 350 deploys and executes a computer program stored in the ROM320 on the RAM 330 to thereby control the information processingapparatus 300. The CPU 350 realizes each function of a generation unit360 and a second input/output unit 370. The generation unit 360generates presentation information to be displayed on the head mounteddisplay 100 by executing a presentation information generating processto be described below. The generation unit 360 of the embodiment has thefollowing two modes as processing modes in the presentation informationgenerating process.

(1) First mode in which common presentation information is generated

(2) Second mode in which plural pieces of presentation information aregenerated to respectively correspond to the head mounted displays 100 ato 100 n

The second input/output unit 370 as a presentation unit includes aplurality of second input/output units 371 to 37 n. Similarly to thefirst input/output unit 310, the second input/output units 371 to 37 nare input/output interfaces conforming to various communicationstandards. Since the detail is the same as that of the firstinput/output unit 310, the descriptions thereof will be omitted.

In the embodiment, the second input/output unit 371 and the secondinput/output unit 372 are input/output interfaces conforming to thewireless LAN. The second input/output unit 373 is an input/outputinterface conforming to the USB. The second input/output unit 37 n is aninput/output interface conforming to the DVI. In FIG. 2, the headmounted display 100 a as the image display apparatus is connected to thesecond input/output unit 371. Similarly, the head mounted display 100 bis connected to the second input/output unit 372, the head mounteddisplay 100 c is connected to the second input/output unit 373, and thehead mounted display 100 n is connected to the second input/output unit37 n. In this way, the second input/output unit 370 is configured tohave the second input/output units 371 to 37 n conforming to thedifferent types of communication standards. For this reason, it ispossible to connect the head mounted displays conforming to variouscommunication standards and the second input/output unit 370, and it ispossible to improve the versatility of the information processingapparatus 300.

The external storage apparatus 380 is configured to have a ROM, a RAM, aDRAM, a hard disk, a flash memory, and the like which are notillustrated. The external storage apparatus 380 stores a common videospecification 381, a management table 382, invisible part information383 as an invisible part information storage unit, an operation record384, and an operation mode 385.

FIG. 3 is a table illustrating an example of the common videospecification 381. The common video specification 381 is a table thatdefines a standard specification concerning a video signal of thepresentation information. The common video specification 381 is usedwhen the generation unit 360 generates the presentation information inthe presentation information generating process.

The common video specification 381 of the embodiment includes the numberof pixels, an aspect ratio, and a resolution of an image contained inthe video signal, an image compression method, and an image encodingmethod as the specification of the video signal. The common videospecification 381 is determined in advance, and is stored in theexternal storage apparatus 380. When the common video specification 381is determined, the video signal specifications, which are adopted by thehead mounted displays 100 a to 100 n connected to the secondinput/output unit 370, are preferably taken into consideration.

FIG. 4 is a table illustrating an example of the management table 382.The management table 382 as a role storage unit and an inter-apparatusrelationship storage unit is a table for managing the information of thehead mounted display 100 connected to the second input/output unit 370,more specifically, for managing the information of the head mounteddisplays 100 a to 100 n that are respectively connected to the secondinput/output units 371 to 37 n. The management table 382 is used whenthe generation unit 360 generates the presentation information in thepresentation information generating process, and is also used when thesecond input/output units 371 to 37 n transmit the presentationinformation to the head mounted displays 100 a to 100 n, respectively.

An identifier, an inter-apparatus relationship, a role, and the videospecification are stored in the management table 382 of the embodimentin association with each other. The “identifier” stores each uniqueidentifier for individually identifying the head mounted displays 100connected to the second input/output units 370. The “inter-apparatusrelationship” stores a string of letters indicating master/slaverelationships between the plurality of head mounted displays 100 a to100 n which are connected to the second input/output units 370. In theembodiment, one head mounted display 100 is set to be a master, and theother head mounted displays 100 are set to be slaves. The “role” storesa string of letters indicating a role that the user of the head mounteddisplay 100 identified by the identifier takes in a medical location.The “video specification” stores a string of letters indicating apattern of the specifications (the number of pixels, the aspect ratio,the resolution of the image contained in the video signal, the imagecompression method, the image encoding method, and the like) of thevideo signal in the head mounted display 100 that is identified by theidentifier.

In the example illustrated in FIG. 4, the user of the head mounteddisplay 100 identified by an identifier “1” is an “operating surgeon”,and the video specification of the head mounted display 100 is a“specification A” (an entry E01). Similarly, the user of the headmounted display 100 identified by an identifier “n” is a “nurse” and thevideo specification of the head mounted display 100 is a “specificationE” (an entry E0 n). In the example illustrated in FIG. 4, the headmounted display 100 identified by the identifier “1” is a master, andthe head mounted displays 100 identified by the other identifiers areslaves (entries E01 to E0 n).

The management table 382 may be determined in advance and be stored inthe external storage apparatus 380. Alternatively, the management table382 may be automatically generated in an initial process when the headmounted display 100 is connected to the information processing apparatus300. Specifically, for example, when the generation unit 360 detects theconnection of the head mounted display 100 and the second input/outputunit 370, the generation unit 360 gives a unique identifier to the headmounted display 100, and acquires a designation of master/slave, a roleof the user, and a video specification from the head mounted display100. The generation unit 360 stores the given identifier, thedesignation of master/slave, the acquired role, and the acquired videospecification in the management table 382. The initial process isperformed whenever the head mounted displays 100 a to 100 n arerespectively connected to the second input/output units 370, and thusthe generation unit 360 can automatically generate the management table382.

As described above, the “inter-apparatus relationship” in the managementtable 382 is acquired from the head mounted display 100. In addition,for example, the “inter-apparatus relationship” may be determined by thefollowing methods.

-   -   The “inter-apparatus relationship” is determined based on the        role that is acquired from the head mounted display 100. For        example, the generation unit 360 sets the head mounted display        100 of the operating surgeon to a master, and sets the head        mounted displays 100 of users other than the operating surgeon        to slaves.    -   The “inter-apparatus relationship” is determined based on        acquisition of current position information of the head mounted        display 100 and a positional relationship between the head        mounted displays. For example, the generation unit 360 acquires        the current position information of the head mounted display        100. The generation unit 360 sets the head mounted display 100        positioned closest to a center of an operating table to a        master, and sets other head mounted displays 100 to slaves.    -   The “inter-apparatus relationship” is determined based on        differences of hardware and software configurations of the head        mounted display 100. For example, the generation unit 360        acquires a model number of the head mounted display 100. The        generation unit 360 sets the head mounted display 100 with a        specific model number to a master, and sets other head mounted        displays 100 to slaves.

As described above, the “role” in the management table 382 is acquiredfrom the head mounted display 100. In addition, for example, the “role”may be determined by the following methods.

-   -   The “role” is determined based on current position information        of the head mounted display 100. For example, the external        storage apparatus 380 of the information processing apparatus        300 pre-stores layout information that contains layout of        equipment (the operating table and the medical apparatus 500) in        the operation room, and layout of standing positions of staffs        based on the roles in the operation room. The generation unit        360 acquires the current position information of the head        mounted display 100, and compares the acquired current position        information with the layout information, and thus the role of        the user of the head mounted display 100 is determined.    -   The “role” is determined based on an image (an outside scene        image) in a direction of a visual field of the user of the head        mounted display 100. The generation unit 360 acquires an outside        scene image from the head mounted display 100, and recognizes        the outside scene image. Accordingly, the generation unit 360        specifies the type of the medical apparatus 500, the type of the        equipment in the operation room, a part of the patient body, and        the like which the user sees. The generation unit 360 determines        the role of the user of the head mounted display 100 based on        the specified information.

The invisible part information 383 (the invisible part informationstorage unit) illustrated in FIG. 2 stores invisible part informationthat does not appear in an external appearance of an object. Theinvisible part information is image information or letter informationillustrating a structure of the object. When the object is a livingbody, the invisible part information is, for example, a CT image, an MRIimage, an X-ray image, a fluoroscopic image, an ultrasonic diagnosisimage, an endoscopic image, or a thermographic image, and letterinformation incident to the image. The invisible part information isphotographed in advance, and is stored in the invisible part information383.

The operation record 384 illustrated in FIG. 2 is a recording unit thatrecords the medical apparatus information acquired by the informationprocessing apparatus 300 and the presentation information generated bythe information processing apparatus 300 in association with a date andtime when the information is acquired or generated. The medicalapparatus information acquired by the information processing apparatus300 indicates the medical apparatus information that the informationprocessing apparatus 300 acquires from the plurality of medicalapparatuses 501 to 50 n which are respectively connected to the firstinput/output units 311 to 31 n. The presentation information generatedby the information processing apparatus 300 indicates the entirepresentation information that the generation unit 360 generates in thepresentation information generating process. The record of theinformation acquired and generated by the information processingapparatus 300 can be regarded to be the substantially same as a log ofan operation (a diagnosis, a treatment, or a curing). Accordingly, inthis manner, after the operation or the like is ended, it is possible touse the log of the operation in the analysis of content of theoperation, training and academic purpose, and the like.

The operation mode 385 illustrated in FIG. 2 stores a processing mode(that is, an operation mode of the generation unit 360) in thepresentation information generating process of the generation unit 360.Content of the operation mode 385 can be changed by an input devicewhich the information processing apparatus 300 includes and which is notillustrated. The content of the operation mode 385 can be also changedby a change instruction from the head mounted display 100.

A-3. Configuration of Head Mounted Display Apparatus:

FIG. 5 is a block diagram illustrating a functional configuration of thehead mounted display 100 a. Hereinafter, the head mounted display 100 ais illustrated and described as an example of a configuration of thehead mounted display 100. The head mounted display 100 a is an opticaltransmission type head mounted display apparatus on which a user can seea virtual image and concurrently, can directly see an outside scene. Thepresentation information acquired from the information processingapparatus 300 is displayed as a virtual image on the head mounteddisplay 100 a. The head mounted display 100 a includes an image displayunit 20 on which the user sees the virtual image in a state where theuser wears the head mounted display 100 a on the head, and a controlunit 10 that controls the image display unit 20. Various signals aretransmitted between the image display unit 20 and the control unit 10via a cable 40.

A-3-1. Configuration of Control Unit:

The control unit 10 includes the following units: an input informationacquisition unit 110 that acquires an operation input to an input devicesuch as a touch pad, a cross key, a foot switch, a gesture, or amicrophone; a storage unit 120 configured to have a ROM, a hard disk,and the like; a power supply 130 that supplies an electrical power toeach unit of the head mounted display 100 a; a wireless communicationunit 132 that wirelessly communicates with other devices conforming to apredetermined wireless communication standard such as the wireless LAN,Bluetooth, or the like; a current position acquisition unit 134; a CPU140; an interface 180 that connects the information processing apparatus300 and various external devices OA; and transmitting units (Tx) 51 and52. The units are connected to each other via a bus which is notillustrated. The current position acquisition unit 134 detects a currentposition of the user of the head mounted display 100 a by receiving asignal from a GPS satellite, and generates current position informationindicating the current position of the user. The current positionacquisition unit 134 may generate the current position information byreceiving radio waves from a plurality of base stations and bydetermining strength of the received radio waves. For example, thecurrent position information can be realized by a coordinate indicatinga latitude/longitude. Similar to the first input/output unit 310 of theinformation processing apparatus 300, the interface 180 can conform tovarious communication standards.

The CPU 140 reads out and executes a computer program stored in thestorage unit 120, and thus functions as the following units: anoperating system (OS) 150; an image processing unit 160; a voiceprocessing unit 170 that supplies voice signals to speakers of a rightearphone 32 and a left earphone 34; and a display control unit 190. Theimage processing unit 160 generates a signal based on a video signal ofthe presentation information that is input via the interface 180 as thepresentation information acquisition unit or the wireless communicationunit 132. The image processing unit 160 supplies the generated signal tothe image display unit 20 via the transmitting units 51 and 52 and thecable 40, and thus controls the display on the image display unit 20.The signal supplied to the image display unit 20 can be a analog signalor a digital signal which are widely known. The display control unit 190generates a control signal to control a right display driving unit 22and a left display driving unit 24 of the image display unit 20, and thecontrol signal is transmitted via the transmitting units 51 and 52. Thecontrol signal is an individual signal through which a right LCD controlunit 211 turns on or off the driving of a right LCD 241, through which aright backlight control unit 201 turns on or off the driving of a rightbacklight 221, through which a left LCD control unit 212 turns on or offthe driving of a left LCD 242, and through which a left backlightcontrol unit 202 turns on or off the driving of a left backlight 222.

A-3-2. Configuration of Image Display Unit:

The image display unit 20 is a mounting body that the user wears on thehead. In the embodiment, the image display unit 20 has an eyeglassshape. The image display unit 20 includes the right display driving unit22 and the left display driving unit 24 that generate and emit imagelight for an image; a right optical image display unit 26 and a leftoptical image display unit 28 (refer to FIG. 2) that guide the imagelight to both eyes of the user; a camera 61 as the image acquisitionunit, and a 9-axis sensor 66.

The right display driving unit 22 includes a receiving unit (Rx) 53; theright backlight (BL) control unit 201 and the right backlight (BL) 221which function as light sources; the right LCD (Liquid Crystal Display)control unit 211 and the right LCD 241 which function as a displayelement; and a right projection optical system 251. The right backlightcontrol unit 201 drives the right backlight 221 based on a controlsignal that is input via the receiving unit 53. The right backlight 221is a light emitting body such as an LED or an electroluminescence (EL).The right LCD control unit 211 drives the right LCD 241 based on aninput video signal. The right LCD 241 is a transmissive liquid crystalpanel on which a plurality of pixels are arranged in a matrix shape. Theright LCD 241 drives liquid crystal at a position of each of the pixelsthat are arranged in a matrix shape. Accordingly, the right LCD 241changes transmittance of light that transmits through the right LCD 241,and modulates illumination light irradiated from the right backlight 221to effective image light for the image. The right projection opticalsystem 251 is configured to have a collimate lens that converts theimage light emitted from the right LCD 241 into parallel light fluxes.The left display driving unit 24 has the same configuration as that ofthe right display driving unit 22 and operates similar to the rightdisplay driving unit 22.

The right optical image display unit 26 and the left optical imagedisplay unit 28 are arranged to be positioned in front of the right andleft eyes of the user when the user wears the image display unit 20(refer to FIG. 2). The right optical image display unit 26 includes aright light guide plate 261 formed of a light transmissive resinmaterial or the like. The right light guide plate 261 reflects the imagelight along a predetermined optical path to guide the image light outputfrom the right display driving unit 22 to a right eye RE of the user. Adiffraction grating or a semi-transmissive reflective film may be usedin the right light guide plate 261. The left optical image display unit28 has the same configuration as that of the right optical image displayunit 26, and operates similar to the right optical image display unit26. In this way, the image light guided to both eyes of the user of thehead mounted display 100 a is focused on the retinas of the user, andthus the user can see the virtual image. The user of the head mounteddisplay 100 a of the embodiment can concurrently see the virtual imageand the outside scene in the background of the virtual image in aportion of the visual field where the virtual image is displayed. In aportion of the visual field where the virtual image is not displayed,the user can directly see the outside scene through the right opticalimage display unit 26 and the left optical image display unit 28.

The 9-axis sensor 66 is a motion sensor that detects acceleration (3axes), angular velocities (3 axes), and terrestrial magnetism (3 axes).When the user wears the image display unit 20 on the head, the 9-axissensor 66 functions as a movement detecting unit that detects a movementof the head of the user of the head mounted display 100 a. Here, themovement of the head includes changes in velocity, acceleration, angularvelocity, direction, and orientation of the head. The orientation of thehead is also referred to as “orientation information”. The camera 61 asa visual field image acquisition unit is arranged at a position thatcorresponds to the middle of the forehead of the user when the userwears the image display unit 20. The camera 61 photographs an image ofthe outside scene in a front direction of the image display unit 20, inother words, in the direction of the visual field of the user in a statewhere the user wears the head mounted display 100 a, and the camera 61acquires the outside scene image. The camera 61 is a so-called visiblelight camera. The camera 61 of the embodiment is a monocular camera, buta stereo camera may be adopted.

A-4. Presentation Information Generating Process:

FIG. 6 is a flow chart illustrating procedures of the presentationinformation generating process. The presentation information generatingprocess is a process of generating the presentation information to bedisplayed on the head mounted display 100, and is executed by thegeneration unit 360 of the information processing apparatus 300 (referto FIG. 2). The presentation information generating process is dividedinto the first mode in which the common presentation information isgenerated, and the second mode in which the plural pieces ofpresentation information are generated to respectively correspond to thehead mounted displays 100 a to 100 n. The process executed in the secondmode can be roughly divided into an initial process of steps S120 toS134 and an individual process of steps S136 to S142.

The generation unit 360 confirms an operation mode of the generationunit 360 with reference to content that is stored in the operation mode385 (step S100).

A-4-1. First Mode:

When the operation mode is the “first mode” (step S100: the first mode),the generation unit 360 retrieves the master head mounted display 100from the management table 382 (step S110). Specifically, the generationunit 360 retrieves an entry, the “inter-apparatus relationship” of whichis stored as a master, with reference to the management table 382, andthe generation unit 360 acquires an “identifier” of the retrieved entry.

The generation unit 360 acquires a request for an information sourceselection from the master head mounted display 100 (step S112).Specifically, the generation unit 360 requests the head mounted display100 with the identifier acquired in step S110 to transmit an request forthe information source selection. Here, for example, the request for aninformation source selection indicates any one of requests listed in thefollowing a1 to a6.

(a1) A request for reading the medical apparatus information acquired bythe medical apparatus 500

(a2) A request for placing a telephone call to the doctor PR in theexternal facility

(a3) A request for reading data recorded in the in-hospital database OD

(a4) A request for reading data recorded in the invisible partinformation 383

(a5) A request for reading data recorded in the operation record 384

(a6) A request for reading the medical apparatus information of themedical apparatus 500, the display unit of which the user of the headmounted display 100 can hardly see

The information source in the request a1 is the medical apparatus 500.Similarly, the information source in the request a2 is the doctor PR;the information source in the request a3 is the database OD; theinformation source in the request a4 is the invisible part information383; the information source in the request a5 is the operation record384; and the information source in the request a6 is the medicalapparatus 500, the display unit of which the user can hardly see. Here,for example, “the medical apparatus 500, the display unit of which theuser can hardly see” is a general term of the following medicalapparatus 500.

-   -   The medical apparatus 500, the display unit (a display or the        like) of which the user cannot see or can hardly see because the        medical apparatus 500 is present at a blind spot of the user.    -   The medical apparatus 500, the display unit of which the user        cannot see or can hardly see because the medical apparatus 500        is present in a place remote from the user.    -   The medical apparatus 500, the display unit of which the user        cannot see or can hardly see because the medical apparatus 500        is not present at a blind spot of the user and is present in a        place close to the user, but a visual line of the user faces the        display unit.

The requests a1 to a6 may be individually issued, and may be combinedtogether. However, in the embodiment, a request for information sourceacquisition includes at least the request a1. The request for aninformation source selection is appropriately issued based on anoperation of the user of the head mounted display 100.

The head mounted display 100 can transmit the request for an informationsource selection to the information processing apparatus 300, forexample, as per the following procedures b1 to b5.

(b1) The user of the head mounted display 100 inputs a desiredinformation source and information for specifying information in theinformation source via the input device (a touch pad, a cross key, afoot switch, a gesture, a microphone, or the like) of the head mounteddisplay 100. If a voice input via the microphone is adopted as an inputmethod, it is possible to considerably improve convenience of the userwhen it is difficult for the user to operate the head mounted display100 by the hands in a medical location. In the requests a1 and a6, it isnot necessary to input the “information for specifying information inthe information source”.

(b2) The input information acquisition unit 110 of the head mounteddisplay 100 acquires a signal based on an operation input of the userand transmits the signal to the OS 150.

(b3) The OS 150 of the head mounted display 100 generates a request foran information source selection. Specifically, in the request a1, the OS150 generates the request for an information source selection, whichincludes an identifier of the head mounted display 100 and an identifierfor specifying the information source designated in the procedure b1. Inthe requests a2 to a5, the OS 150 generates a request for an informationsource selection, which includes the identifier of the head mounteddisplay 100, the identifier for specifying the information sourcedesignated in the procedure b1, and the information for specifying theinformation in the information source. In the request a6, the OS 150generates a request for an information source selection which includesthe identifier of the head mounted display 100, and current positioninformation and orientation information of the head mounted display 100.The current position information and the orientation information areused to specify “the medical apparatus 500, the display unit of whichthe user can hardly see” in the request a6.

(b4) The OS 150 of the head mounted display 100 transmits the requestfor an information source selection generated in the procedure b3 to theinformation processing apparatus 300 to which the OS 150 is connectedvia the interface 180 and the wireless communication unit 132.

(b5) The second input/output unit 370 of the information processingapparatus 300 transmits the received request for an information sourceselection to the generation unit 360.

The generation unit 360 acquires the information for specifying theinformation source and the information for specifying the information inthe information source, which are contained in the request for aninformation source selection in step S114 of in the presentationinformation generating process (refer to FIG. 6). Furthermore, when thecurrent position information and the orientation information areinserted as the information for specifying the information source, thegeneration unit 360 compares the current position information and theorientation information with position information acquired from theentirety of other medical apparatuses 500, and the generation unit 360specifies “the medical apparatus 500, the display unit of which the usercan hardly see”. In addition, in step S114, the generation unit 360 mayspecify “the medical apparatus 500, the display unit of which the usercan hardly see” by combining the current position information and theorientation information with a movement (changes in velocity,acceleration, angular velocity, direction, and orientation) of the headof the user that is obtained by the 9-axis sensor 66 of the head mounteddisplay 100.

The generation unit 360 accesses the designated (specified) informationsource to acquire the designated information. The generation unit 360generates the common presentation information supplied to the entiretyof the head mounted displays 100 connected to the second input/outputunits 370 using the acquired information. The common presentationinformation is referred to as “presentation information PN*”. Thegeneration unit 360 stores the generated presentation information PN* inthe operation record 384 (step S116).

FIGS. 7A and 7B illustrate views describing a method of generating thepresentation information. When single information is acquired in stepS116, the generation unit 360 generates presentation information PN byconverting an image contained in acquired information CP1 as per thevideo signal specification defined in the common video specification 381(refer to FIG. 7A). In particular, when an aspect ratio of an imagecontained in the acquired information CP1 is different from the aspectratio of an image defined in the common video specification 381 duringthe conversion, the generation unit 360 enlarges or reduces the imagewhile maintaining the aspect ratio of an image contained in the acquiredinformation CP1. Then, a margin (indicated by a hatched line in FIG. 7A)is formed in a longitudinal direction or a lateral direction of thepresentation information PN.

The generation unit 360 executes any one of the following procedures c1and c2 with respect to the margin. Accordingly, the margin is displayedas a virtual image on the head mounted display 100, and thus it ispossible to suppress the blockage of the visual field of the user.

(c1) The generation unit 360 inserts black dummy dot data in the marginof the presentation information PN.

(c2) The generation unit 360 sets a signal (an enable signal) to a Lovalue (an invalid value) with respect the margin of the presentationinformation PN, and the signal is intended to perform the switchingbetween validity and invalidity of generation of image light in theright display driving unit 22 and the left display driving unit 24 ofthe head mounted display 100.

In this manner, the aspect ratio of an image contained in thepresentation information PN does not change. It is preferable tostrictly maintain an aspect ratio of an image photographed by themedical apparatus 500 in such a manner that the user see an actualunspoiled size of the image (for example, a still image and a movingimage, a CT image, an MRI image, or an X-ray image of an operated part)in a medical location where the information processing apparatus 300 isused. In this regard, if the aspect ratio of an image which is containedin the medical apparatus information CP1 acquired in step S116 ismaintained, and the information source is the medical apparatus 500, itis possible to suppress unexpected distortion of the medical apparatusinformation measured or photographed by the medical apparatus 500. FIG.7B illustrates presentation information PNx that is distorted becausethe aspect ratio of an image contained in the acquired information CP1is not maintained, and the image is enlarged to the aspect ratio of animage defined in the common video specification 381.

FIG. 8 is a view describing the method of generating the presentationinformation. When plural pieces of information are acquired in stepS116, the generation unit 360 generates a first image in which the imagecontained in the acquired entire information (in examples illustrated inFIGS. 10A and 10B, medical apparatus information CP2 and CP3) isarranged. At this time, the generation unit 360 maintains the aspectratio of an image contained in the acquired information. The imagecontained in the acquired information is preferably arranged in an endportion of the first image. The generation unit 360 generates thepresentation information PN by converting the generated first image asper the video signal specification defined in the common videospecification 381 (refer to FIG. 8). The generation unit 360 executesany procedure of the methods c1 and c2 with respect to a margin(indicated by a hatched line in FIG. 8) in which the image is notarranged, in the first image. Accordingly, the margin is displayed as avirtual image on the head mounted display 100, and it is possible tosuppress the blockage of the visual field of the user.

FIGS. 9A and 9B illustrate views showing a first example of thepresentation information. FIG. 9A is an example of the presentationinformation PN generated by the generation unit 360 when an X-ray image(CP4) of a patient stored in the invisible part information 383 isdesignated in the request for an information source selection. The samepresentation information as that illustrated in FIG. 9A is generatedwhen the medical apparatus 500 (an X-ray image photographing apparatus)from which the invisible part information can be acquired is designatedin the request for an information source selection.

FIG. 9B is an example of the presentation information PN generated bythe generation unit 360 when a timer for surgery is designated as themedical apparatus 500, a pulse rate meter is designated as the medicalapparatus 500, and a biological monitor is designated as the medicalapparatus 500 in the request for an information source selection. Thefollowing is arranged in each end portion of the presentationinformation PN: present time CP5 acquired from the timer for surgery,time CP7 elapsed from the start of surgery, a pulse image CP6 acquiredfrom the pulse rate meter, and an image CP8 of a heart rate (HR), ablood pressure (SYS/DIA), and arterial oxygen saturation (SpO2) acquiredfrom the biological monitor.

FIGS. 10A and 10B illustrates views showing a second example of thepresentation information. FIG. 10A is an example of the presentationinformation PN generated by the generation unit 360 when an academicbook (CP9) stored in the in-hospital database OD and operation procedureinformation (CP10) stored in the in-hospital database OD are designatedin the request for an information source selection. FIG. 10B is anexample of the presentation information PN generated by the generationunit 360 when a visible light camera for photographing a still image anda moving image of an operated part is designated as the medicalapparatus 500 and a telephone call with the doctor PR in the externalfacility is designated in the request for an information sourceselection.

FIG. 11 is a view illustrating a third example of the presentationinformation. FIG. 11 is an example of the presentation information PNgenerated by the generation unit 360 when the biological monitor isdesignated as the medical apparatus 500 in the request for aninformation source selection. The biological monitor illustrated in FIG.11 does not include an interface that can be connected to the firstinput/output unit 310. However, when the information processingapparatus 300 is configured to include the visible light camera, and thevisible light camera photographs a screen (CP13) of the biologicalmonitor, as illustrated in FIG. 11, the generation unit 360 can generatethe presentation information PN using the medical apparatus informationfrom the medical apparatus that does not include an interface that canbe connected to the first input/output unit 310.

When the presentation information PN is generated, the head mounteddisplay 100 may directly or indirectly designate “arrangement”, a “size”and “transmittance” of the information of each information source in therequest for an information source selection. Here, the “directdesignation” means that the head mounted display 100 designatesnumerical values indicating a coordinate at which each information isarranged, and a size and transmittance of each information. The“indirect designation” means that the head mounted display 100designates a desired mode from display modes that are prepared inadvance, for example, from an “outside scene priority mode” in which anoutside scene is prioritized, an “image priority mode” in which an imageis prioritized, and the like. For example, when the outside scenepriority mode is designated, as illustrated in FIG. 8, the generationunit 360 generates the presentation information PN in such a manner thatthe information is arranged in a small size at an end portion of ascreen so as not to interfere with the visual field of the user at thecenter of the screen. When the image priority mode is designated, asillustrated in FIG. 9A, the generation unit 360 generates thepresentation information PN in such a manner that the information isarranged in a large size at the center of the screen.

In step S118 of the presentation information generating process (referto FIG. 6), the generation unit 360 transmits the generated presentationinformation PN* to the entirety of the second input/output units 370(the second input/output units 371 to 37 n). The second input/outputunit 371 converts the image contained in the received presentationinformation PN* as per the video signal specification of the headmounted display 100 a. At this time, the second input/output unit 371may refer to the “video specification” from the management table 382.After the conversion is performed as per the video specification, thesecond input/output unit 371 transmits the presentation information PN*to the head mounted display 100 a. Similarly, the second input/outputunits 372 to 37 n convert images contained in the received presentationinformation PN* as per the video signal specifications of the headmounted displays 100 b to 100 n, respectively. The second input/outputunits 372 to 37 n transmit the converted presentation information PN* tothe head mounted displays 100 b to 100 n, respectively.

FIG. 12 is a view illustrating a state in which the presentationinformation PN* is respectively displayed as a virtual image on theplurality of head mounted displays 100 a and 100 b. The aforementioneddisplay processing is performed by the image processing unit 160 of thehead mounted display 100 a that receives the presentation informationPN* transmitted from the second input/output unit 371 via the interface180 or the wireless communication unit 132. As a result, image lightguided to both eyes of the user of the head mounted display 100 a isfocused on the retinas of the user, and thus the user of the headmounted display 100 a can see the presentation information PN* as avirtual image VI in a visual field VR. The user can directly see apatient lying on the operating table (an outside scene SC) through theright optical image display unit 26 and the left optical image displayunit 28. Similarly, the aforementioned display processing is performedby the image processing unit 160 of the head mounted display 100 b thatreceives the presentation information PN* transmitted from the secondinput/output unit 372, and the user of the head mounted display 100 bcan see the presentation information PN* as the virtual image VI in thevisual field VR. In the example illustrated in FIG. 12, the medicalapparatus information CP acquired from the medical apparatus 502 forphotographing a still image and a moving image of an operated part isdisplayed in the virtual image VI (PN*) seen by the user of the headmounted display 100 a and the user of the head mounted display 100 b.

FIG. 13 is a view illustrating a state in which the presentationinformation PN* is displayed as virtual images on the plurality of headmounted displays 100 a and 100 b that are positioned in differentplaces. As illustrated in FIG. 13, the second input/output units 370(the second input/output units 371 to 37 n) of the informationprocessing apparatus 300 may perform a fitting process in such a mannerthat an orientation and a size of the image contained in thepresentation information coincide with an orientation and a size of atarget object contained in the outside scene SC. For example, the targetobject is the body of a patient or a part of the body in a medicallocation. Hereinafter, the fitting process will be described withreference to the second input/output unit 371. In step S118 of thepresentation information generating process (refer to FIG. 6), thesecond input/output unit 371 converts the image contained in thepresentation information PN* as per the video signal specification ofthe head mounted display 100 a, and then performs the followingprocedures d1 to d5.

(d1) The second input/output unit 371 instructs the head mounted display100 a to photograph an outside scene image using the camera 61, andacquires the photographed outside scene image.

(d2) The second input/output unit 371 extracts characteristics of thetarget object contained in the outside scene image using the followingimage recognition methods 1 and 2. The methods 1 and 2 may be combined.

1. An edge (characteristics) of the target object is detected

2. A marker (characteristics) which is attached to the target object inadvance is detected. Various types of markers can be used as the markerattached to the object. For example, a tape, a seal, a magic marker, alaser marker, and a magic tape (registered trademark) can be used. Thenumber of markers attached to the target object is arbitrarily adjusted.

(d3) The second input/output unit 371 extracts the characteristics ofthe target object by recognizing the image contained in the receivedpresentation information. At this time, the second input/output unit 371can use the same image recognition methods as the aforementioned methods1 and 2.

(d4) The second input/output unit 371 compensates the image CP containedin the presentation information PN* in such a manner that thecharacteristics of the outside scene image extracted in the procedure d2are positioned in the same place in which the characteristics of thepresentation information extracted in the procedure d3 are positioned.Specifically, when the method 1 is used in the procedures d2 and d3, thesecond input/output unit 371 at least enlarges, reduces, rotates,reverses, trims, or distorts the image CP contained in the presentationinformation PN*, or removes noise from the image CP in such a mannerthat a contour and a characteristic portion of the edge of the outsidescene image are positionally aligned with a contour and a characteristicportion of the edge of the presentation information. Here, when thetarget object is a living body, the characteristic portion is a contourof an incised skin, a contour of an internal organ, a joint, distal endsof limbs, a blood vessel, a bone, and the like. In contrast, when themethod 2 is used in the procedures d2 and d3, the second input/outputunit 371 at least enlarges, reduces, rotates, reverses, trims, ordistorts the image CP contained in the presentation information PN*, orremoves noise from the image CP in such a manner that the marker of theoutside scene image is positionally aligned with the marker of thepresentation information.

In this manner, the second input/output units 370 (the secondinput/output units 371 to 37 n) recognize the image in the direction ofthe visual field of the user of the head mounted display 100 (the headmounted display apparatus), and thus the second input/output units 370extract the characteristics of the object contained in the outside sceneand compensates the presentation information generated by the generationunit 360 based on the extracted characteristics. For this reason, whenthe presentation information contains the invisible part informationsuch as a CT image, an MRI image, or an X-ray image, it is possible toalign the invisible part information with the object (for example, thebody of a patient) contained in the outside scene. As a result, it ispossible to improve convenience of the user of the head mounted display100. Since the fitting processing is executed in the second input/outputunit 370 connected to the specific head mounted display 100, aprocessing load is not applied to the second input/output units 370connected to the other head mounted displays 100. Furthermore, since thehead mounted display 100 can receive the presentation information thatundergoes the fitting processing, the image processing unit 160 of thehead mounted display 100 may display the received presentationinformation as it is. As a result, it is possible to simplify theconfiguration of the head mounted display 100.

As described above, when the operation mode of the generation unit 360is the “first mode”, the generation unit 360 generates the commonpresentation information PN* that contains at least the medicalapparatus information selected by the user of the head mounted display100 (that is, the specific image display apparatus) in which theinter-apparatus relationship of the management table 382 is a “master”.The second input/output units 370 (presentation units) output thegenerated common presentation information PN* to the plurality of headmounted displays 100 a to 100 n (the image display apparatuses) whichare respectively connected to the second input/output units 371 to 37 n.As a result, the information processing apparatus 300 can controldisplay on the plurality of head mounted displays 100 a to 100 n basedon the selection of the user of the specific head mounted display 100.

A-4-2. Second Mode (Initial Process):

When the operation mode is the “second mode” in step S100 of thepresentation information generating process (refer to FIG. 6) (stepS100: the second mode), the generation unit 360 sets a variable i usedin the presentation information generating process to “1” (step S120).Thereafter, the generation unit 360 refers to an i-th entry of themanagement table 382 (step S122). The generation unit 360 acquires an“identifier” and a “role” which are stored in the i-th entry of themanagement table 382 (step S124).

Hereinafter, the head mounted display 100 having the identifier acquiredin step S124 is also referred to as a “head mounted display 100 i”. Thehead mounted display 100 i indicates the head mounted display 100 thatis currently under the processing in the initial process. For example,when the variable i is 1, the head mounted display 100 i indicates thehead mounted display 100 a of the operating surgeon, which has theidentifier of 1 and is stored in the first entry E01 of the managementtable 382.

In step S126 of the presentation information generating process (referto FIG. 6), the generation unit 360 specifies the medical apparatus 500based on the role of the user of the head mounted display 100 i.Specifically, when the generation unit 360 carries out the “role”acquired in step S124, the generation unit 360 specifies the medicalapparatus 500 that the user of the head mounted display 100 i has tocheck. The medical apparatus 500 is specified based on a practice in amedical location. For example, when the role is an operating surgeon, anapparatus which acquires the invisible part information such as a CTimage or an X-ray image is specified as the medical apparatus 500. Whenthe role is an anesthesiologist, each of a blood pressure meter, anelectrocardiogram, an airway pressure meter, a ventilation meter, acapnogram and a pulse oximeter is specified as the medical apparatus500. The information processing apparatus 300 may store a table in theexternal storage apparatus 380 in advance, in which the role and themedical apparatus 500 necessary for the role are stored in associationwith each other.

After the medical apparatus 500 is specified, the generation unit 360generates the presentation information PNi to be transmitted to the headmounted display 100 i using the medical apparatus information that isacquired from the medical apparatus 500 specified in step S126 among themedical apparatus information of the medical apparatuses 501 to 50 n,which the first input/output units 311 to 31 n acquire in step S128. Thegeneration unit 360 generates the presentation information in the samemethod as described with reference to FIGS. 7A, 7B, and 8. Thereafter,the generation unit 360 stores the generated presentation informationPNi in the operation record 384.

In step S130, the generation unit 360 transmits the presentationinformation PNi generated in step S124 to a second input/output unit 370i to which the head mounted display 100 i is connected. The secondinput/output unit 370 i converts an image contained in the receivedpresentation information PNi as per a video signal specification of thehead mounted display 100 i. At this time, the second input/output unit370 i may execute the fitting process. The detail is the same asdescribed in step S118.

FIG. 14 is a view illustrating a state in which the presentationinformation PNi is displayed as virtual images on the plurality of headmounted displays 100 a and 100 b after steps S122 to S134 are executedtwo times. The aforementioned display processing is performed by theimage processing unit 160 of the head mounted display 100 i thatreceives the presentation information PNi via the interface 180 or thewireless communication unit 132. As a result, the user of the headmounted display 100 i can see the presentation information PNi as avisual image, which contains the medical apparatus information acquiredfrom the medical apparatus 500 based on the role of the user. In anexample illustrated in FIG. 14, the medical apparatus information CP4acquired from the medical apparatus 500, which an operating surgeon hasto check, is displayed as a virtual image VI (i) that the user of thehead mounted display 100 a sees. The medical apparatus information CP2and CP3 acquired from the medical apparatus 500, which a first assistanthas to check, is displayed as the virtual image VI (i) that the user ofthe head mounted display 100 b sees.

The generation unit 360 increments the variable i in step S132 of thepresentation information generating process (refer to FIG. 6).Thereafter, the generation unit 360 determines whether the variable i isequal to or greater than n (step S134). The fact that the variable i issmaller than n (step S134: NO) means that a part of the head mounteddisplays 100 do not end the initial process among the head mounteddisplays 100 which are respectively connected to the second input/outputunits 370. For this reason, the generation unit 360 transfers theprocess to step S122 to continuously execute the initial process. Incontrast, the fact that the variable i is equal to or greater than n(step S134: YES) means that the initial processes for the entirety ofthe head mounted displays 100 are ended. For this reason, the generationunit 360 transfers the process to an individual process in step S136.

As described above, when the operation mode of the generation unit 360is the “second mode”, the generation unit 360 repeatedly executes thesteps S122 to S134 in the initial process. Accordingly, the generationunit 360 generates plural pieces of the presentation information PNithat contain at least the medical apparatus information which isacquired from the medical apparatuses 500 based on the respective rolesof the users of the plurality of head mounted displays 100 i (100 a to100 n). In addition, the second input/output units 370 (the presentationunits) sequentially output the plural pieces of generated presentationinformation PNi to the plurality of head mounted displays 100 a to 100 n(the image display apparatuses) which are respectively connected to thesecond input/output units 371 to 37 n. As a result, the informationprocessing apparatus 300 can individually control display on theplurality of head mounted displays 100 a to 100 n based on the role ofeach user of the head mounted display 100, and it is possible to improveconvenience of each user of the head mounted display 100.

A-4-3. Second Mode (Individual Process):

In the individual process, based on a request from the head mounteddisplay 100, the generation unit 360 generates the presentationinformation that contains the medical apparatus information acquiredfrom the requested medical apparatus 500 and the information acquiredfrom information sources other than the requested medical apparatus 500.

The generation unit 360 determines whether a request for an informationsource selection is received from the head mounted display 100 in stepS136 of the presentation information generating process (refer to FIG.6). The request for an information source selection is the same as therequests a1 to a6 which are described in step S112. When the request foran information source selection is not received (step S136: notreceived), the generation unit 360 transfers the process to the stepS136, and waits for the request for an information source selection.

In contrast, when the request for an information source selection isreceived (step S136: received). The generation unit 360 acquires anidentifier of the head mounted display 100 i contained in the requestfor an information source selection, information for specifying theinformation source, and information for specifying the information inthe information source (step S128). Furthermore, when the currentposition information and the orientation information are inserted as theinformation for specifying the information source, the generation unit360 compares the current position information and the orientationinformation with position information acquired from the entirety ofother medical apparatuses 500, and the generation unit 360 specifies“the medical apparatus 500, the display unit of which the user canhardly see”. Here, similarly to in step S114, the generation unit 360may combine the current position information and the orientationinformation with a movement of the head of the user of the head mounteddisplay 100. The head mounted display 100 i indicates the head mounteddisplay 100 (that is, the head mounted display 100 that is currentlyunder the processing) which is a source of issuing the request for aninformation source selection.

The generation unit 360 accesses the designated information source, andacquires the designated information in step S140. The generation unit360 generates the presentation information PNi for the head mounteddisplay 100 i using the acquired information, and stores the generatedpresentation information PNi in the operation record 384. The generationunit 360 generates the presentation information in the same method asdescribed with reference to FIGS. 7A, 7B, and 8. Thereafter, thegeneration unit 360 stores the generated presentation information PNi inthe operation record 384.

In step S142, the generation unit 360 transmits the presentationinformation PNi generated in step S140 to the second input/output unit370 i to which the head mounted display 100 i is connected. The secondinput/output unit 370 i converts an image contained in the receivedpresentation information PNi as per the video signal specification ofthe head mounted display 100 i. At this time, the second input/outputunit 370 i may execute the fitting process. The detail is the same asdescribed in step S118.

The aforementioned display processing is performed by the imageprocessing unit 160 of the head mounted display 100 i that receives thepresentation information PNi via the interface 180 or the wirelesscommunication unit 132. As a result, the user of the head mounteddisplay 100 i, which transmits the request for an information sourceselection, can see a virtual image of the presentation information PNithat includes the information acquired from the information sourcedesignated in the request for an information source selection.

As described above, when the operation mode of the generation unit 360is the “second mode”, the generation unit 360 executes the individualprocess. Accordingly, the generation unit 360 generates the pluralpieces of presentation information PNi that contain at least the medicalapparatus information which is selected by each user of the plurality ofhead mounted displays 100 i (100 a to 100 n). The second input/outputunits 370 (the presentation units) sequentially output the plural piecesof generated presentation information PNi to the plurality of headmounted displays 100 a to 100 n (the image display apparatuses) whichare respectively connected to the second input/output units 371 to 37 n.As a result, the information processing apparatus 300 can individuallycontrol display on the plurality of head mounted displays 100 a to 100 nbased on the role of each user of the head mounted display 100.

As described above, in the information processing apparatus 300 of FirstEmbodiment, the generation unit 360 generates the presentationinformation PN using the plural pieces of medical apparatus informationacquired by the first input/output units 310 (acquisition units and thefirst input/output units 311 to 31 n), and thus the informationprocessing apparatus 300 can unify the information (the medicalapparatus information) of various medical apparatuses 500 (the medicalapparatuses 501 to 50 n) which are connected to the first input/outputunits 310 of the information processing apparatus 300. The secondinput/output units 370 (the presentation units) output the generatedpresentation information PN to the plurality of head mounted displays100 a to 100 n (the image display apparatuses), respectively, and thusthe second input/output units 370 can control display on the pluralityof head mounted displays 100 a to 100 n, respectively. As a result, itis possible to unify the information of various medical apparatuses 500,and it is possible to realize the information processing apparatus 300that can control the display on the plurality of head mounted displays100 a to 100 n.

Furthermore, the generation unit 360 has the first mode in which thecommon presentation information PN* is generated from the unifiedmedical apparatus information, which is common to the plurality of headmounted displays 100 a to 100 n (the image display apparatuses), and thesecond mode in which the plural pieces of presentation information PNiare generated to respectively correspond to the plurality of headmounted displays 100 a to 100 n. For this reason, the generation unit360 uses the modes separately, and thus the information processingapparatus 300 can control the display on the plurality of head mounteddisplays 100 a to 100 n.

As described above, the information processing system 1000 of FirstEmbodiment can unify the information (the medical apparatus information)of various medical apparatuses 500 (the medical apparatuses 501 to 50 n)in the information processing apparatus 300, and it is possible torealize the information processing system 1000 that can control thedisplay on the plurality of head mounted displays 100 a to 100 n (thehead mounted display apparatuses).

A-5. Additional Process:

In the presentation information generating process (refer to FIG. 6),the following additional processes may be further executed. Theadditional processes may be added independently or may be combinedtogether.

A-5-1. Additional Process 1:

In an additional process 1, the information processing apparatus 300performs a color conversion process for the presentation information.Specifically, the second input/output unit 370, which receives thepresentation information from the generation unit 360, causes the camera61 of the head mounted display 100 to acquire an outside scene image.The second input/output unit 370 carries out the color conversionprocess so as to improve visibility of the image contained in thepresentation information based on brightness, contrast, chroma, andcolor of the acquired outside scene image. For example, the secondinput/output unit 370 can perform the color conversion process in whichthe color of the image contained in the presentation information isconverted into a complementary color of the color of the outside sceneimage.

In this manner, the additional process 1 can further improve visibilityof the virtual image VI for the user of the head mounted display 100.

A-5-2. Additional Process 2:

An additional process 2 enables the information processing apparatus 300to realize a so-called stabilizer function that suppresses theflickering of the virtual image VI incident to a small shake of the headof the user of the head mounted display 100. Specifically, the secondinput/output unit 370, which receives the presentation information fromthe generation unit 360, repeatedly performs the fitting processdescribed in the procedures d1 to d4, and the transmission of thepresentation information described in steps S118, S130, and S142 of thepresentation information generating process at every predetermined time.Accordingly, the following is repeatedly performed at everypredetermined time: the photographing of the outside scene image, andthe compensation and the display of the image contained in thepresentation information. As a result, the presentation information canbe displayed as the virtual image VI so as to trace the movement overtime of the head of the user.

However, when the presentation information traces a slight movement or afluctuation of the head of the user of the head mounted display 100, andthus the presentation information is frequently changed, there is aproblem in that eyestrain of the user is induced and concentration ofthe user is disturbed. Therefore, the second input/output unit 370 mayperform processes described in the following procedures e1 and e2between the procedure d1 and the procedure d2.

(e1) The second input/output unit 370 calculates the amount of change inRGB histogram between the outside scene image photographed in theprocedure c1 and the outside scene image photographed in the procedured1 in the fitting process that is executed at the previous cycle.

(e2) When the amount of change calculated in the procedure f1 exceeds apredetermined threshold value, the second input/output unit 370continuously performs the processes subsequent to the procedure d2.

In this manner, in the additional process 2, when there is a smallamount of change between the outside scene image in the fitting processexecuted at the previous cycle and the outside scene image in thefitting process currently being executed, that is, when the movement ofthe head of the user of the head mounted display 100 is very small, thesecond input/output unit 370 does not compensate the image contained inthe presentation information. As a result, since the presentationinformation can be prevented from tracing a very small movement or afluctuation of the head of the user of the head mounted display 100, andcan be prevented from frequently being changed, it is possible tosuppress eyestrain or deterioration of concentration of the user. Aconfiguration may be adopted in which the user can turn on and off thestabilizer function.

A-5-3. Additional Process 3:

In an additional process 3, the information processing apparatus 300changes a size of the image, which is contained in the presentationinformation and is displayed as the virtual image VI, based on adistance between the image display unit 20 of the head mounted display100 and an object that is present in the direction of the visual fieldof the user. Specifically, the second input/output unit 370, whichreceives the presentation information from the generation unit 360, canperform processes described in the following procedures f1 to f3 insteadof the fitting process described in the procedures d1 to d4. When thisprocess is carried out, the head mounted display 100 is configured tofurther include a distance measurement sensor. The distance measurementsensor is a sensor that uses reflected light to acquire the distancebetween the object present in the direction of the visual field of theuser and the image display unit 20. For example, the distancemeasurement sensor can be arranged in the vicinity of the camera 61.

(f1) The second input/output unit 370 periodically acquires ameasurement value of the distance measurement sensor of the head mounteddisplay 100.

(f2) The second input/output unit 370 determines an enlargement ratio(or a reduction ratio) based on the acquired measurement value of thedistance measurement sensor.

(f3) The second input/output unit 370 at least enlarges, reduces,rotates, reverses, trims, or distorts the image contained in thepresentation information, or removes noise from the image using thedetermined enlargement or reduction ratio.

In this manner, according to the additional process 3, the secondinput/output unit 370 can change the size of the presentationinformation to be displayed as the virtual image VI based on thedistance between the object present in the direction of the visual fieldof the user of the head mounted display 100 and the image display unit20. For this reason, it is possible to improve convenience of each userof the head mounted display 100.

B. Second Embodiment

In a second embodiment of the invention, a configuration will bedescribed in which the generation unit 360 can generate combinedinformation. Hereinafter, only portions having configurations andoperations different from those of First Embodiment will be described.In the drawings, the same reference signs as those of First Embodimentwill be assigned to portions with the same configurations as those ofFirst Embodiment. The detailed description will be omitted.

B-1. Configuration of Information Processing System:

A schematic configuration of an information processing system 1000 aaccording to Second Embodiment is the substantially same as that ofFirst Embodiment illustrated in FIG. 1. However, the informationprocessing system 1000 a of Second Embodiment includes an informationprocessing apparatus 300 a instead of the information processingapparatus 300.

B-2. Configuration of Information Processing Apparatus:

FIG. 15 is a block diagram illustrating a functional configuration ofthe information processing apparatus 300 a according to SecondEmbodiment. Second Embodiment is different from First Embodimentillustrated in FIG. 2 in that the information processing apparatus 300 aincludes a generation unit 360 a instead of the generation unit 360.When the operation mode is the “second mode”, the generation unit 360 aexecutes a combined information generating process to be described laterin parallel with the presentation information generating process. Thecombined information generating process is a process in which thegeneration unit 360 a generates the combined information obtained bycombining together the presentation information of the entirety of thehead mounted displays 100 (the head mounted displays 100 a to 100 n)that are respectively connected to the second input/output units 370.

B-3. Configuration of Head Mounted Display Apparatus:

The schematic configuration of the head mounted display 100 according toSecond Embodiment is the same as that of First Embodiment illustrated inFIG. 5.

B-4. Presentation Information Generating Process:

Procedures of the presentation information generating process accordingto Second Embodiment is the same as that of First Embodiment illustratedin FIG. 6.

B-5. Additional Process:

Even in the presentation information generating process of SecondEmbodiment, the same additional processes as those of First Embodimentmay be executed as described in the “A-5-1 Addition Process 1” to the“A-5-3 Additional Process 3”.

B-6. Combine Information Generating Process:

FIG. 16 is a flow chart illustrating procedures of the combinedinformation generating process. The combined information generatingprocess is a process in which the combined information is generated tobe displayed on the head mounted display 100. The generation unit 360 aof the information processing apparatus 300 a (refer to FIG. 15)executes the combined information generating process. In step S100 ofthe presentation information generating process (refer to FIG. 6), whenthe operation mode is the “second mode” (step S100: the second mode),the combined information generating process is executed in parallel withthe initial process and the individual process described in FIG. 6.

The generation unit 360 a determines whether a request for combinedinformation is received from the head mounted display 100 (step S200).Here, the request for combined information is a request for transmissionof the combined information, and is transmitted along with an identifierof the requesting head mounted display 100. When the request forcombined information is not received (step S200: not received), thegeneration unit 360 a transfers the process to step S200, and waits toreceive the request for combined information.

In contrast, when the request for combined information is received (stepS200: received), the generation unit 360 a acquires an identifier of therequesting head mounted display 100 i, which is contained in the requestfor combined information (step S202). Here, the head mounted display 100i indicates the head mounted display 100 (that is, the head mounteddisplay 100 that is currently under the processing) which is a source ofissuing the request for combined information.

The generation unit 360 a accesses the operation record 384 in stepS204. The generation unit 360 a acquires the latest presentationinformation which the generation unit 360 a generates for the entiretyof the head mounted displays 100 (the head mounted displays 100 a to 100n) that are respectively connected to the second input/output units 370.The latest presentation information can be determined based on “a dateand time of the information generation” that are stored in the operationrecord 384 in association with the presentation information. Here, thepresentation information of the requesting head mounted display 100 imay not be acquired. The generation unit 360 a generates the combinedinformation by combining together plural pieces of the acquiredpresentation information.

FIG. 17 is a view describing a method of generating the combinedinformation. The generation unit 360 a performs processes illustrated inthe following procedures g1 to g4 on the plural pieces of acquiredpresentation information.

(g1) The generation unit 360 a determines a reduction ratio of thepresentation information based on a total number of the acquiredpresentation information and resolution that is stored in the commonvideo specification 381.

(g2) The generation unit 360 a reduces the image contained in the i-thpresentation information (the presentation information PNi) at thereduction ratio determined in the procedure g1.

(g3) The generation unit 360 a arranges the image reduced in theprocedure g2 at a position at which the reduced image does not overlapwith other images, and the reduced image is overwritten and saved ascombined information JN.

(g4) The generation unit 360 a increments the variable i, and repeatedlyperforms the procedures g2 and g3 until “i is equal to or larger than atotal number of the acquired presentation information” is met.

(g5) The generation unit 360 a executes the same procedures as describedin c1 and c2 with respect to a margin (indicated by a hatched line inFIG. 17) in which the image for the presentation information among thecombined information JN is not arranged. Dummy dot data is inserted inthe margin, or an enable signal is adjusted.

(g6) The generation unit 360 a converts the image contained in thegenerated combined information JN as per the video signal specificationdefined in the common video specification 381.

FIG. 17 illustrates an example of the combined information JN in whichsix presentation information PN1 to PN6 generated in this manner arerespectively arranged at positions at which six presentation informationdoes not overlap with each other.

In step S206 of the combined information generating process (refer toFIG. 16), the generation unit 360 a transmits the combined informationJN generated in step S204 to the second input/output unit 370 i to whichthe head mounted display 100 i is connected, and then transfers theprocess to step S200. The second input/output unit 370 i converts theimage contained in the received presentation information PNi as per thevideo signal specification of the head mounted display 100 i.

FIG. 18 is a view illustrating a state in which the combined informationJN is displayed as a virtual image on the head mounted display 100 i.The aforementioned display process is performed by the image processingunit 160 of the head mounted display 100 i that receives the combinedinformation JN transmitted from the second input/output unit 370 i viathe interface 180 or the wireless communication unit 132. As a result,image light guided to both eyes of the user of the head mounted display100 i is focused on the retinas of the user, and thus the user of thehead mounted display 100 i can see the combined information JN as thevirtual image VI in the visual field VR. In the example illustrated inFIG. 18, the combined information JN of the presentation information PN1to PN6 for the respective head mounted displays 100 a to 100 f isdisplayed as the virtual image VI in the visual field VR of the user ofthe head mounted display 100 g. Due to limitation of space, thepresentation information PN3 to PN6 is not illustrated in FIG. 18.

As described above, the generation unit 360 a of Second Embodimentcombines the plural pieces of presentation information PNi generated inthe presentation information generating process to generate the combinedinformation JN. The second input/output unit 370 (the presentation unit)outputs the generated combined information JN to the specific headmounted display 100 i (the image display apparatus) that issues therequest for combined information. As a result, the user of the specifichead mounted display 100 i can understand what presentation informationPN is displayed on the plurality of head mounted displays 100 a to 100 n(the image display apparatuses) which are connected to the informationprocessing apparatus 300. In addition, it is possible to improveconvenience of the user of the specific head mounted display 100 i.

C. Third Embodiment

A third embodiment of the invention describes a configuration in whichthe user of the head mounted display is guided through a series ofpredetermined steps, and in which the guiding steps can be changed inreal time based on the medical apparatus information. Hereinafter, onlyportions having configurations and operations different from those ofFirst Embodiment will be described. In the drawings, the same referencesigns as those of First Embodiment will be assigned to portions with thesame configurations as those of First Embodiment described above. Thedetailed description will be omitted.

C-1. Configuration of Information Processing System:

A schematic configuration of an information processing system 1000 baccording to Third Embodiment is the substantially same as that ofSecond Embodiment illustrated in FIG. 1. However, the informationprocessing system 1000 b of Second Embodiment includes an informationprocessing apparatus 300 b instead of the information processingapparatus 300.

C-2. Configuration of Information Processing Apparatus:

FIG. 19 is a block diagram illustrating a functional configuration ofthe information processing apparatus 300 b according to ThirdEmbodiment. Third Embodiment is different from First Embodimentillustrated in FIG. 2 in that the information processing apparatus 300 bincludes a CPU 350 b instead of the CPU 350, and includes an externalstorage apparatus 380 b instead of the external storage apparatus 380.The CPU 350 b includes a generation unit 360 b instead of the generationunit 360, and further includes a step management unit 365.

Processes in the presentation information generating process executed bythe generation unit 360 b are partially different from those of FirstEmbodiment illustrated in FIG. 6. The step management unit 365 executesa step management process to be described later in parallel with thepresentation information generating process executed by the generationunit 360. The step management process is a process in which the stepmanagement unit 365 transmits to the generation unit 360 b a series ofsteps through which the user of the head mounted display 100 is guided,and in which the steps are changed based on a state of a patientestimated from the medical apparatus information.

An external storage apparatus 380 b further includes step information386 and emergency response information 387 in addition to all theinformation described in First Embodiment.

FIG. 20 a schematic diagram illustrating an example of the stepinformation 386. The step information 386 in the embodiment is a table(or a database configured to have a plurality of tables) in which aseries of steps for each of various operations are stored in advance. Inthe step management process, the step information 386 is referenced soas to guide the user of the head mounted display 100 through the steps.For illustrative purposes, FIG. 20 illustrates only a single operation,and a specific name of operation, a name of operative method, andcontent of steps will not be illustrated. A name for specifying anoperation is stored in the “name of operation”. A name for specifying anoperative method is stored in the “name of operative method”. Since anoperation may be performed in a plurality of different operativemethods, steps for the operation are specified based on the combinationof the name of operation and the operative method. Content of series ofsteps specified by the name of operation and the operative method aresequentially stored in the “step”. Since a part of steps for anoperation are differently executed based on symptoms and a state of apatient, the branched steps are present in the example illustrated inFIG. 20. In the presentation information generating process (refer toFIG. 6), when the second mode is realized to generate the plural piecesof presentation information, the step information 386 is configured toinclude a table and the like, in which a series of steps are stored foreach role in various operations.

FIG. 21 is a table illustrating an example of the emergency responseinformation 387. The emergency response information 387 of theembodiment stores operative method, conditions, symptoms, and emergencyresponses in association with each other in advance. In the stepmanagement process, the emergency response information 387 is referencedso as to change steps through which the user of the head mounted display100 is guided. A name for specifying an operative method is stored in an“operative method”. The operative method is used so as to be associatedwith the “operative method” in the step information 386. Conditions forjudging the medical apparatus information is stored in the “condition”.A name is stored in the “symptom”, and the name indicates symptoms thatare estimated to occur in a patient when the medical apparatusinformation satisfies the conditions stored in the “condition”.Treatments for resolving or mitigating the symptoms of the patient arestored in the “emergency response”.

C-3. Configuration of Head Mounted Display Apparatus:

The schematic configuration of the head mounted display 100 according toThird Embodiment is the same as that of First Embodiment illustrated inFIG. 5.

C-4. Presentation Information Generating Process:

Procedures of the presentation information generating process accordingto Third Embodiment are the same as those of First Embodimentillustrated in FIG. 6 except for the following points to be describedbelow. A case where the presentation information generating process isexecuted in the first mode in which the common presentation informationis generated will be described. In step S114 illustrated in FIG. 6, thegeneration unit 360 b receives a “request (a7) for an operationprocedure guide” as a request for an information source selection fromthe head mounted display 100. Thereafter, the generation unit 360 brequests the step management unit 365 to execute the step managementprocess. The request a7 may be issued independently or may be combinedwith the requests a1 to a6 of First Embodiment. In step S116, thegeneration unit 360 b receives steps from the step management unit 365,and generates the presentation information PN* using the receivedinformation.

A case where the presentation information generating process is executedin the second mode in which the plural pieces of presentationinformation are generated will be described. In step S126 illustrated inFIG. 6, the generation unit 360 b transmits a role of the user of thehead mounted display 100 i to the step management unit 365. Thereafter,the generation unit 360 b transmits the role of the user to the stepmanagement unit 365, and requests the step management unit 365 toexecute the step management process. In step S128, the generation unit360 b receives steps from the step management unit 365 based on the roleof the user of the head mounted display 100 i, and generates thepresentation information PNi using the received information.

C-5. Additional Process:

Even in the presentation information generating process of ThirdEmbodiment, the same additional processes as those of First Embodimentmay be executed as described in the “A-5-1 Addition Process 1” to the“A-5-3 Additional Process 3”.

C-6. Step Management Process:

FIG. 22 is a flow chart illustrating procedures of the step managementprocess. The step management process is a process in which the stepmanagement unit transmits to the generation unit 360 b steps throughwhich the user of the head mounted display 100 is guided, and in whichthe steps are changed based on a state of a patient estimated from themedical apparatus information.

In step S300, the step management unit 365 acquires a series ofpredetermined steps with reference to the step information 386. Here,“the series of predetermined steps” mean a set of series of steps(indicated by solid line frame L1 in FIG. 20) which an operatingsurgeon, other doctors, or the like determines among the combinations ofoperations, the operative methods, and the steps stored in the stepinformation 386 in advance. The series of predetermined steps may end inthe middle of series of steps stored in the step information 386(indicated by broken line frame L2 in FIG. 20). For example, when adoctor preferably selects the best steps while performing an operationin practicality and monitoring a state of the patient, the stepsindicated by broken line frame L2 are scheduled.

When the presentation information generating process is executed in thefirst mode (in other words, when the generation unit 360 b instructsthat only the step management process be executed), and the steps basedon the role of the user is stored in the step information 386, in stepS300, the step management unit 365 acquires “a series of predeterminedsteps” for the role of the user of the head mounted display 100, theinter-apparatus relationship of which is a “master” in the managementtable 382 (refer to FIG. 4). In contrast, when the presentationinformation generating process is executed in the second mode (in otherwords, the role of the user and the execution of the step managementprocess are instructed by the generation unit 360), in step S300, thestep management unit 365 acquires “the series of predetermined steps”for the role of the user, which are received from the step information386.

In step S302, the step management unit 365 acquires a single step fromthe series of steps acquired in step S300, and transmits the single stepto the generation unit 360 b. The generation unit 360 b which receivesthe single step generates the presentation information PN* or thepresentation information PNi for a guide to the single step in thepresentation information generating process, and the generation unit 360b transmits the presentation information PN* or the presentationinformation PNi to the head mounted display 100.

The step management unit 365 checks a state of the patient in step S304.Specifically, the step management unit 365 acquires the entire medicalapparatus information measured by the medical apparatus, which measuresa state of the patient, via the first input/output units 310 (theacquisition units, and the first input/output units 311 to 31 n).

In step S306, the step management unit 365 determines whether a rapidchange occurs in the patient. Specifically, the step management unit 365determines whether the entire medical apparatus information received instep S304 satisfies the “condition” stored in the emergency responseinformation 387 in step S304. When the entire medical apparatusinformation does not satisfy the condition (step S306: NO), the stepmanagement unit 365 transfers the process to step S314. In contrast,when at least a single piece of medical apparatus information satisfiesthe condition (step S306: YES), the step management unit 365 transfersthe process to step S308.

In step S308, the step management unit 365 acquires an emergencyresponse based on a state of the patient with reference to the emergencyresponse information 387. Specifically, the step management unit 365refers to an entry, the medical apparatus information of which satisfiesthe “condition” among the emergency response information 387, and thestep management unit 365 refers to the “emergency response” for theentry.

The step management unit 365 transmits content of the emergency responseto the generation unit 360 b in step S310. The aforementioned processesare executed in the same manner when the presentation informationgenerating process is executed not only in the first mode but also inthe second mode. The generation unit 360 b which receives the emergencyresponse generates the presentation information PN* or the presentationinformation PNi for a guide to the emergency response in thepresentation information generating process, and transmits thepresentation information PN* or the presentation information PNi to thehead mounted display 100.

The step management unit 365 checks a state of the patient in step S312.The detail is the same as illustrated in step S304. A time period, forwhich a state of the patient is assumed to be stabilized by theemergency response, is preferably placed between step S310 and stepS312.

In step S314, the step management unit 365 determines whether the seriesof steps being currently guided to the user are appropriate.Specifically, the step management unit 365 determines whether the entiremedical apparatus information received in step S312 satisfies the“condition” stored in the emergency response information 387. When theentire medical apparatus information does not satisfy the condition(step S314: YES), the step management unit 365 estimates that the seriesof steps being currently guided to the user are appropriate, and thestep management unit 365 transfers the process to step S302. Thereafter,in step S302, the step management unit 365 acquires the next step fromthe series of steps acquired in step S300, and transmits the next stepto the generation unit 360 b. In addition, the step management unit 365checks a state of the patient, and repeatedly executes theaforementioned processes.

In this manner, the step management unit 365 repeatedly acquires theplural pieces of medical apparatus information, and changes the seriesof steps to be transmitted to the generation unit based on the pluralpieces of latest medical apparatus information. For this reason, theinformation processing apparatus 300 b can change a series of steps tobe displayed on the plurality of head mounted displays 100 a to 100 n inreal time based on the plural pieces of medical apparatus informationthat are acquired from the plurality of medical apparatuses.

In contrast, when at least one piece of medical apparatus informationsatisfies the condition (step S306: YES), the step management unit 365estimates that the series of steps being currently guided to the userare appropriate, and transfers the process to step S316. In step S316,the step management unit 365 transmits to the generation unit 360 b aguide that a state of the patient is bad, and thus the series of stepsbeing currently guided to the user are assumed not to be appropriate.The aforementioned process is executed in the same manner when thepresentation information generating process is executed not only in thefirst mode but also in the second mode. When the generation unit 360 breceives the guide that the series of steps being currently guided tothe user are not appropriate, the generation unit 360 b generates thepresentation information PN* or the presentation information PNi so asto guide the user to the fact that the series of steps are notappropriate, and the generation unit 360 b transmits the presentationinformation PN* or the presentation information PNi to the head mounteddisplay 100 in the presentation information generating process.Thereafter, the step management unit 365 stops transmitting thesubsequent steps, and ends the process.

In step S310, the step management unit 365 adds a “new step” to “theseries of predetermined steps” by transmitting the emergency responsebased on the acquired medical apparatus information to the generationunit 360 b. However, in step S310, the step management unit 365 mayexecute a “removal of part of steps” or a “replacement of part of steps”with respect to “the series of predetermined steps”. For example, whenthe step to be removed or the step to be replaced is stored as the“emergency response” in the emergency response information 387, theremoval of part of steps or the replacement of part of steps can berealized. In the embodiment, the following is collectively referred toas “a change of series of steps” the addition of a new step (the removalof part of steps, or the replacement of part of steps) illustrated instep S310, and the stop of transmission of steps illustrated in stepS316.

As described above, in the information processing apparatus 300 b ofThird Embodiment, the step management unit 365 transmits to thegeneration unit 360 b the information on a series of predeterminedsteps, and the generation unit 360 b generates the presentationinformation PN* and PNi which contain the received steps. For thisreason, the information processing apparatus 300 b can display a seriesof predetermined steps on the plurality of head mounted displays 100 ato 100 n (the image display apparatuses) connected to the informationprocessing apparatus 300 b. In addition, the step management unit 365changes a series of steps to be transmitted based on plural pieces ofmedical apparatus information acquired by the first input/output units311 to 31 n (the acquisition units). For this reason, the informationprocessing apparatus 300 b can change a series of steps to be displayedon the plurality of head mounted displays 100 a to 100 n based on theplural pieces of medical apparatus information acquired from theplurality of medical apparatuses. As a result, it is possible to providethe information processing apparatus 300 b that can flexibly change theseries of predetermined steps based on the medical apparatusinformation, and can guide the users of the head mounted displays 100 ato 100 n through the steps.

Furthermore, in the first mode, the step management unit 365 of theinformation processing apparatus 300 b of Third Embodiment transmits, tothe generation unit 360 b, a series of steps for the user of the headmounted display 100 (that is, the specific image display apparatus), theinter-apparatus relationship of which is a “master” in the managementtable 382. For this reason, the information processing apparatus 300 bcan display “a series of common steps for the user of the specific imagedisplay apparatus” on the plurality of head mounted displays 100 a to100 n (the image display apparatuses) in the first mode. In the secondmode, the step management unit 365 transmits a series of steps to thegeneration unit 360 b based on a role of each user of the plurality ofhead mounted displays 100 a to 100 n. For this reason, the informationprocessing apparatus 300 b can individually display “the series of stepsbased on the role of each user of the image display apparatus” on theplurality of head mounted displays 100 a to 100 n in the second mode.

D. Modification Examples

In each embodiment described above, a part of the configuration realizedby hardware may be replaced with software. In contrary, a part of theconfiguration realized by software may be replaced with hardware. Inaddition, the following modifications can be also made to theembodiments.

Modification Example 1

The configuration of the information processing system is illustrated ineach embodiment described above. However, the configuration of theinformation processing system can be arbitrarily determined insofar asthe arbitrarily determined configuration does not depart from the spiritof the invention. For example, it is possible to add, remove, or convertthe apparatuses of the information processing system. It is possible tochange a configuration of a network of the apparatuses of theinformation processing system.

For example, the management table, the invisible part information, andthe operation record of the information processing apparatus may bestored in the database OD connected to the in-hospital LAN. Theprovision of the database OD is not limited to the hospital, and thedatabase OD may be provided on the outside (for example, a cloud serverthat is provided outside the hospital) via the Internet INT.

Modification Example 2

The configuration of the information processing apparatus is illustratedin each embodiment described above. However, the configuration of theinformation processing apparatus can be arbitrarily determined insofaras the arbitrarily determined configuration does not depart from thespirit of the invention. For example, it is possible to add, remove, orconvert each configuration part.

For example, the information processing apparatus may include variousinput devices such as a touch panel, an input button, a keyboard, amouse, a microphone, and a camera. Similarly, the information processingapparatus may include various output devices such as a display, anindicator, and a printer so as to output a simple log on an operation.

For example, in each embodiment, the CPU deploys a computer programstored on the ROM or the hard disk on the RAM, and executes the computerprogram. Accordingly, each function of the generation unit and thesecond input/output unit (the presentation units) is realized. However,the functions may be realized using an ASIC (Application SpecificIntegrated Circuit) that is designed so as to realize the functions.

For example, the information processing apparatus is configured toinclude a display. In addition, the generation unit of the informationprocessing apparatus may display the combined information generated inthe combined information generating process on the display of theinformation processing apparatus. In this manner, a user in the vicinityof the information processing apparatus can understand content displayedon the entirety of the head mounted displays (the image displayapparatuses) at a glance, and it is convenient.

Modification Example 3

The configuration of the head mounted display is illustrated in eachembodiment described above. However, the configuration of the headmounted display can be arbitrarily determined insofar as the arbitrarilydetermined configuration does not depart from the spirit of theinvention. For example, it is possible to add, remove, or convert eachconfiguration part. For example, the control unit and the image displayunit may be connected via a wireless signal transmission path of awireless LAN, infrared ray communication, Bluetooth, or the like. Forexample, a primary battery, a fuel cell, a solar cell, a thermal cell,or the like may be adopted as the power supply.

In each embodiment described above, the allocation of the configurationelements to the control unit and the image display unit is only anexample, and can be adopted in various forms. For example, the followingforms may be adopted. (i) A form in which the control unit is equippedwith processing functions such as a CPU and a memory, and the imagedisplay unit is equipped with only a display function; (ii) A form inwhich both control unit and image display unit are equipped withprocessing functions such as a CPU and a memory; (iii) A form in whichthe control unit and the image display unit are integrated together (forexample, a form in which the image display unit contains the controlunit, and functions as an eyeglass type wearable computer); (iv) A formin which a smart phone or a portable game machine is used instead of thecontrol unit; and (v) A form in which the control unit and the imagedisplay unit are configured to be wirelessly communicable and wirelesslychargeable, and thus a cable is removed.

For example, the input information acquisition unit may acquire anoperation input from the user using various methods other than themethods illustrated in the embodiments. For example, the inputinformation acquisition unit may acquire an operation input via a footswitch (a switch that is operated by the feet of the user). For example,the image display unit may be provided with a visual line detecting unitsuch as an infrared sensor, and a visual line of the user may bedetected. Accordingly, the input information acquisition unit mayacquire an operation input via a command associated with a movement ofthe visual line. For example, a camera may be used to detect a gestureof the user, and the input information acquisition unit may acquire anoperation input via a command associated with the gesture. At thedetection of the gesture, it is possible to mark a fingertip of theuser, a finger ring attached to the hands of the user, a medicalinstrument or the like held by the hands of the user for the detectionof a movement. When the input information acquisition unit can acquirean operation input via the foot switch or the visual line, the inputinformation acquisition unit can easily acquire the operation input fromthe user even in a medical location where it is difficult for the userto free up the hands.

For example, in the embodiments, the head mounted display is thebinocular transmissive head mounted display, but the head mounteddisplay may be a monocular head mounted display. The embodiment may beconfigured to adopt a non-transmissive head mounted display throughwhich an outside scene is prevented from being seen when the user wearsthe head mounted display. The embodiments adopt the head mounteddisplay, the image display unit of which the user wears like eyeglasses.However, the head mounted display may adopt the image display unit withother shape, for example, the image display unit that the user wearslike a hat. An earphone or a headband earphone may be adopted as theearphone, or the earphone may be omitted. Furthermore, for example, theembodiment may be configured to adopt a head-up display (HUD) mounted ona vehicle such as an automobile or an airplane. In addition, theembodiment may adopt the head mounted display that is built in a bodyprotection tool such as a helmet.

FIGS. 23A and 23B illustrates views showing an external configuration ofeach head mounted display in modification examples. In the exampleillustrated in FIG. 23A, an image display unit 20 a includes a rightoptical image display unit 26 a instead of the right optical imagedisplay unit 26, and includes a left optical image display unit 28 ainstead of the left optical image display unit 28. The right opticalimage display unit 26 a is formed to be smaller than the optical memberin First Embodiment, and is obliquely arranged above the right eye of auser when the user wears the head mounted display. Similarly, the leftoptical image display unit 28 a is formed to be smaller than the opticalmember in First Embodiment, and is obliquely arranged above the left eyeof the user when the user wears the head mounted display. In the exampleillustrated in FIG. 23B, an image display unit 20 b includes a rightoptical image display unit 26 b instead of the right optical imagedisplay unit 26, and includes a left optical image display unit 28 binstead of the left optical image display unit 28. The right opticalimage display unit 26 b is formed to be smaller than the optical memberin First Embodiment, and is obliquely arranged below the right eye of auser when the user wears the head mounted display. The left opticalimage display unit 28 b is formed to be smaller than the optical memberin First Embodiment, and is obliquely arranged below the left eye of theuser when the user wears the head mounted display. In this manner, it issatisfactory for the optical image display unit to be arranged in thevicinity of the eyes of the user. A size of the optical member, whichforms the optical image display unit, may be also arbitrarilydetermined. It is possible to realize the head mounted display in such amanner that only parts of the eyes of the user are covered with theoptical image display units, in other words, the eyes of the user arenot completely covered with the optical image display units.

For example, in each embodiment described above, the display drivingunit is configured to include the backlight, the backlight control unit,the LCD, the LCD control unit, and the projection optical system, butthe configuration form is only an example. In addition to or instead ofthe configuration parts, the display driving unit may includeconfiguration parts so as to realize other display driving method. Forexample, the display driving unit may emit image light using a frontlight method or a reflection method. For example, the display drivingunit may be configured to include an organic EL (OrganicElectro-Luminescence) display, an organic EL control unit, and aprojection optical system. For example, the display driving unit canalso use a digital micromirror device or the like instead of the LCD.For example, the invention is also applicable to a laser retinalprojection head mounted display apparatus.

Modification Example 4

The embodiment illustrates an example of the presentation informationgenerating process. However, the procedures of the presentationinformation generating process are only examples, and variousmodifications can be made to the procedures. For example, a part ofsteps may be omitted, or additionally, other steps may be added. Thesteps may be executed in a changed order.

For example, the invisible part information described in the embodimentsis assumed to be a two-dimensional image. However, the medical apparatuscapable of photographing a three-dimensional invisible part image may beconnected. The invisible part information stored in the external storageapparatus may be a three-dimensional model produced from a plurality ofimages photographed by an image photographing device. When thethree-dimensional invisible part information is used, the generationunit renders a three-dimensional model prior to generating thepresentation information. The generation unit generates the presentationinformation using the rendered image. In this manner, it is possible tosee information in various directions, which does not appear in anexternal appearance of a target object.

For example, when the presentation information generating process isexecuted in the first mode, the generation unit may generate pluralpieces of common presentation information. Specifically, for example,the head mounted displays, which are respectively connected to thesecond input/output units, are divided into an m (m is an integer equalto or greater than 1) number of groups. In addition, the generation unitmay generate common presentation information for each group like commonpresentation information for a first group, . . . common presentationinformation for an m-th group.

For example, when the presentation information generating process (theinitial process) is executed in the second mode, the initial process isexecuted in serial using the variable i with respect to the plurality ofhead mounted displays that are respectively connected to the secondinput/output units. However, the initial process may be executed inparallel with reference to the plurality of head mounted displays thatare respectively connected to the second input/output units.

For example, in the presentation information generating process executedin the second mode, either the initial process or the individual processmay be omitted. The initial process may not be executed immediatelyafter the presentation information generating process is started in thesecond mode. The same process as the initial process may beappropriately executed based on a request from a user or otherapplications.

For example, in both first and second modes, the image contained in thepresentation information may be enlarged or reduced based on a requestfrom a user to be displayed as the virtual image VI on the head mounteddisplay. Similarly, the head mounted display may not display the imagecontained in the presentation information based on a request of a user,and the head mounted display may convert a color of the image containedin the presentation information. When a user inputs a request via avoice input method using a microphone, it is possible to considerablyimprove convenience of each user when the head mounted display is usedin a medical location where it is difficult for the user to operate thehead mounted display by the hands.

Modification Example 5

The embodiment illustrates an example of the combined informationgenerating process. However, the procedures of the combined informationgenerating process are only an example, and various modifications can bemade to the procedures. For example, a part of may be omitted, or othersteps may be further added. The steps may be executed in a changedorder.

For example, the generation unit may allow the to-be-generated combinedinformation to contain information other than a reduced image in thepresentation information. For example, the combined information maycontain the following listed information.

-   -   A string of letters or an image for identifying the head mounted        display to which the presentation information is supplied.    -   A string of letters or an image for indicating a managerial        position of a user of the head mounted display to which the        presentation information is supplied.    -   The medical apparatus information of the medical apparatus.    -   Data stored in the in-hospital database.    -   Data stored in the invisible part information.    -   Data stored in the invisible part information.    -   The medical apparatus information of the medical apparatus the        display unit of which it is difficult for the user to see.

For example, the generation unit may receive a request for combinedinformation from only a specific head mounted display. For example, thespecific head mounted display is a master head mounted display, or ahead mounted display a user of which takes a specific role.

Modification Example 6

The embodiment illustrates an example of the step management process.However, the procedures of the step management process are onlyexamples, and various modifications can be made to the procedures. Forexample, a part of steps may be omitted, or other steps may be furtheradded. The steps may be executed in a changed order.

For example, instead of the addition of a new step (the removal of apart of steps or the replacement of a part of steps) illustrated in stepS310, or in addition to the addition, the removal, or the replacement ofa new step, the embodiment may adopt a configuration in which the doctorPR in an external facility can present a second opinion via a telephonecall with the doctor PR (refer to FIG. 19).

For example, any one of the following may be omitted: the addition of anew step (the removal of a part of steps or the replacement of a part ofsteps) illustrated in step S310, and the stop of transmission of stepsillustrated in step S316.

For example, “a series of predetermined steps” are determined by anoperating surgeon, other doctors, or the like. The step management unitmay be configured to suggest a series of steps, which have to be acandidate for “the series of predetermined steps” (if necessary, pluralpairs of series of steps), to a user of the head mounted display. Inthis case, for example, the step management unit can derive thecandidate by retrieving the step information based on the medicalapparatus information that is acquired in advance.

For example, the step management process may handle only one of thepresentation information generating process in the first mode and thepresentation information generating process in the second mode.

Modification Example 7

The embodiments illustrate an example of the common video specification,the management table, the step information, and the emergency responseinformation. However, the detailed descriptions are only examples, andvarious medications can be made. For example, it is possible to add,remove, or change a field (an item). The management table may be dividedinto a plurality of tables, and may be standardized.

For example, voice prints of the users of the head mounted displays maybe stored in the management table instead of the identifiers forindividually identifying the head mounted displays, or along with theidentifiers. Accordingly, the information processing apparatus canacquire the inter-apparatus relationship, the role, and the videospecification of the head mounted display of the user based on the voiceprint of the user acquired via a microphone, and the informationprocessing apparatus can execute the presentation information generatingprocess.

For example, information (for example, IP addresses, MAC addresses, andencryption keys of the head mounted displays) used in communicationbetween the head mounted display and the information processingapparatus may be further stored in the management table in associationwith the identifiers of the head mounted displays. Instead of “the videospecification” that indicates a pattern of the video signalspecification, an actual video specification (that is, the same item asthat stored in the common video specification) adopted in each headmounted display may be stored in the management table.

For example, the management table stores the inter-apparatusrelationship, the role, and the video specification for each identifierof the head mounted display. However, a “user ID” may be assigned toeach user, and the inter-apparatus relationship, the role, and the videospecification may be stored for each user ID in the management table.

For example, the “emergency response”, which is stored in the emergencyresponse information in advance, is a single response. However, aplurality of responses may be stored in the emergency response. The stepmanagement unit may guide the user of the image display apparatusthrough the plurality of responses. In this manner, the user of theimage display apparatus can select an emergency response from theplurality of emergency responses, which is considered to be the mostsuitable based on his or her experience or situation. As a result, it ispossible to further improve convenience of the information processingapparatus.

Modification Example 8

The invention is not limited to the embodiments, the examples, and themodification examples described above. The invention can be realized invarious configurations insofar as the various configurations do notdepart from the spirit of the invention. For example, it is possible toappropriately replace or combine together technical characteristics inthe embodiments, the examples, and the modification examples, whichrespectively correspond to technical characteristics of the aspectsdescribed in the summary of the invention so as to solve a part or theentirety of the problems, or to achieve a part or the entirety of theadvantages. Unless the technical characteristics are described as beingessential in this document, the technical characteristics can beappropriately removed.

REFERENCE SIGNS LIST

10: control unit

20: image display unit

22: right display driving unit

24: left display driving unit

26: right optical image display unit

28: left optical image display unit

32: right earphone

34: left earphone

40: cable

51: transmitting unit

52: transmitting unit

53: receiving unit

54: receiving unit

61: camera (image acquisition unit)

110: input information acquisition unit

100: head mounted display (image display apparatus, head mounted displayapparatus)

120: storage unit

130: power supply

140: CPU

160: image processing unit

170: voice processing unit

180: interface

190: display control unit

201: right backlight control unit

202: left backlight control unit

211: right LCD control unit

212: left LCD control unit

221: right backlight

222: left backlight

241: right LCD

242: left LCD

251: right projection optical system

252: left projection optical system

261: right light guide plate

262: left light guide plate

300: information processing apparatus

310: first input/output unit

311 to 31 n: first input/output unit (acquisition unit)

320: ROM

330: RAM

340: communication interface

350: CPU

360: generation unit

365: step management unit

370: second input/output unit (presentation unit)

371 to 37 n: second input/output unit

380: external storage apparatus

381: common video specification

382: management table (role storage unit and inter-apparatusrelationship storage unit)

383: invisible part information

384: operation record

385: operation mode

386: step information

387: emergency response information

500: medical apparatus

501 to 50 n: medical apparatus

700: display

1000: information processing system

OA: external device

PC: personal computer

OD: database

VI: virtual image

PN: presentation information

JN: combined information

VR: visual field

PR: doctor

MP1: position

UP1: current position

CP1: medical apparatus information

The invention claimed is:
 1. An information processing apparatuscomprising a processor configured to: acquire plural pieces of medicalapparatus information that are plural pieces of information acquiredfrom a medical apparatus; generate plural pieces of presentationinformation that respectively contain at least a part of the medicalapparatus information; acquire a distance measurement between an objectviewed by a user through a personal image display apparatus and thepersonal image display apparatus; determine an enlargement ratio or areduction ratio based on the distance measurement; and output thegenerated presentation information to the personal image displayapparatus, wherein the presentation information includes the medicalapparatus information that has been manipulated by being enlarged orreduced in size based on the enlargement ratio or the reduction ratio.2. The information processing apparatus according to claim 1, whereinthe processor is further configured to manipulate the medical apparatusinformation in order to determine an edge of the object in thepresentation information.
 3. The information processing apparatusaccording to claim 2, wherein the processor is further configured torotate, distort, or reduce noise in the presentation information.
 4. Theinformation processing apparatus according to claim 1, wherein theprocessor is further configured to manipulate the medical apparatusinformation in order to determine one or more markers of the object inthe presentation information.
 5. The information processing apparatusaccording to claim 4, wherein the processor is further configured torotate, distort, or reduce noise in the presentation information.
 6. Theinformation processing apparatus according to claim 1, wherein an aspectratio of the medical apparatus information is maintained when themedical apparatus information is manipulated.
 7. The informationprocessing apparatus according to claim 1, wherein the personal imagedisplay apparatus is a head mounted display apparatus on which the usercan see a virtual image and an outside scene.
 8. The informationprocessing apparatus according to claim 7, wherein the processor isfurther configured to: acquire from the head mounted display apparatusan image in a direction of a visual field of the user that wears thehead mounted display apparatus, extract characteristics of the objectcontained in the outside scene by recognizing the image acquired in thedirection of the visual field, and compensate the presentationinformation based on the extracted characteristics, and output thecompensated presentation information.
 9. The information processingapparatus according to claim 1, wherein the processor is furtherconfigured to: transmit information on a predetermined step and changethe step to be transmitted based on the acquired plural pieces ofmedical apparatus information, and generate the presentation informationthat contains the received step.
 10. The information processingapparatus according to claim 1, wherein: the medical apparatus includesa plurality of medical apparatuses, in a first mode, the presentationinformation corresponds to the medical apparatus information acquiredfrom only a specific medical apparatus of the plurality of medicalapparatus, and in a second mode, the presentation informationcorresponds to the medical apparatus information acquired from eachmedical apparatus of the plurality of medical apparatus.
 11. Theinformation processing apparatus according to claim 1, wherein theprocessor is further configured to perform a color conversion process onthe presentation information by manipulating brightness, contrast,chroma, and/or color of the presentation information.
 12. Theinformation processing apparatus according to claim 1, wherein theprocessor is further configured to: calculate an amount of change in thepresentation information during a predetermined time interval, comparethe amount of change with a predetermined threshold value, and updatethe generated presentation information that is outputted to the personalimage display apparatus if the amount of change is greater than thepredetermined threshold value.
 13. The information processing apparatusaccording to claim 1, the generated presentation information that isoutputted to the personal image display apparatus is a three-dimensionalimage.
 14. The information processing apparatus according to claim 1,wherein the user can see the generated presentation information and anoutside scene on the personal image display apparatus.
 15. Theinformation processing apparatus according to claim 1, wherein the usercan see the generated presentation information but not an outside sceneon the personal image display apparatus.
 16. The information processingapparatus according to claim 1, wherein the generated presentationinformation is output to the personal image display apparatus only afterthe medical apparatus information has been manipulated by being enlargedor reduced in size.
 17. An information processing method comprising:acquiring plural pieces of medical apparatus information that are pluralpieces of information acquired from a medical apparatus; generatingplural pieces of presentation information that respectively contain atleast a part of the medical apparatus information; acquiring a distancemeasurement between an object viewed by a user through a personal imagedisplay apparatus and the personal image display apparatus; determiningan enlargement ratio or a reduction ratio based on the distancemeasurement; manipulating the medical apparatus information in order toenlarge or reduce in size the presentation information based on theenlargement ratio or the reduction ratio; and outputting the generatedpresentation information to the personal image display apparatus. 18.The method according to claim 17, wherein the medical apparatusinformation is manipulated in order to determine an edge of the objectin the presentation information.
 19. The method according to claim 17,wherein the medical apparatus information is manipulated in order todetermine one or more markers of the object in the presentationinformation.
 20. The information processing apparatus according to claim17, wherein the generated presentation information is output to thepersonal image display apparatus only after the medical apparatusinformation has been manipulated by being enlarged or reduced in size.